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ABSTRACT 



This thesis evaluates the Navy’s test and evaluation 
(T&E) process for aircraft weapon systems, identifies prob- 
lems, and proposes a reorganization of the Navy’s T&E 
agencies. Many changes have been made to improve the T&E 
process, but some of these have resulted in costly ineffi- 
ciencies. Contributing problems include: the Navy’s 

’’independent’’ test agency being tasked more on the basis of 
its apparent independence than on its testing expertise, 
lack of actual independence of the many test agencies, 
excessive testing duplication, inadequate testing expertise, 
lack of operational test objectives, inadequate test planning, 
lack of a weapon system T&E school, and numerous T&E manage- 
ment problems. 

The establishment of a Naval T&E Command which is organi- 
zationally separate from the developing agency and the OPNAV 
program sponsor is proposed. The Naval T&E Command would be 
responsible for all T&E conducted by the Navy with the 
exception of that done by Navy Laboratories. The Naval T&E 
Command would have separate divisions responsible for 
development T&E and operational T&E, and the developing 
agency could task the T&E Command to perform any type of T&E. 
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I. INTRODUCTION 



A. GENERAL 

During the past decade all phases of the Department of 
Defense (DOD) weapon system acquisition process have been 
under increasing criticism. Frequent cost growth, coupled 
with poor operational performance of many weapon systems 
when they entered operational service, has been the main 
impetus for the appointment of numerous committees and 
commissions to study the weapon system acquisition process. 
The President's Blue Ribbon Defense Panel, the Government 
Accounting Office, the Congress's Commission on Government 
Procurement, and the Armed Services Committees of the House 
of Representatives and Senate have consistently identified 
test and evaluation as a major problem area. 

The current DOD major system acquisition policy pro- 
hibits the "total package procurement" concept that was in 
vogue in the 1960's. Concurrency between development and 
production is being minimized, and a determination of opera- 
tional suitability, including logistic support requirements, 
is required -before a production decision is made. This 
change in system acquisition philosophy has resulted in 
greater emphasis in test and evaluation. Consequently, 
problems existing in DOD's test and evaluation (T&E) process 
will have an even greater effect on the acquisition of major 
weapon systems than ever before. 
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It has been less than three years since the major change 
to the DOD weapon system acquisition process was implemented 
by DOD Directive 5000.1, entitled "Acquisition of Major 
Defense Systems" [Ref. 1], Unfortunately, reforming the 
weapon system acquisition process is an undertaking which 
does not lend itself to instantaneous change. By now, most 
of the implementing DOD Directives and Navy instructions 
have been written, but changes in the area of test and 
evaluation are still being made and others comtemplated in 
order to more effectively implement the acquisition philos- 
ophy that is embodied in DOD Directive 5000.1. Many problems 
still exist in the Navy’s T&E process, and new problems are 
being created by the implementation of the new DOD acquisi- 
tion philosophy. 

As a result of the criticism that has been directed at 
T&E, many people believe that improved and expanded test and 
evaluation will be the most significant step towards elimi- 
nating the cost growths and performance underruns that have 
been so typical of weapon system development programs . 
Problems do exist in the military’s T&E process that 
adversely affect the weapon system acquisition process. 
However, many of the problems that exist in T&E are sympto- 
matic of problems existing throughout the weapon system 
acquisition environment, i.e. Congress, DOD, and industry. 

In order to improve the weapon system acquisition process 
attention must be devoted to the entire acquisition process 
and not merely to making patchwork corrections to each 
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segment of the process [Ref. 2], Although changes are 
required to the T&E process, one should not erroneously 
believe that these changes alone will have a significant effect 
in solving the many problems facing the weapon system 
acquisition process today. 

B. PURPOSE 

The purpose of this thesis is to: 

(1) evaluate the Navy’s test and evaluation process for 
aircraft weapon systems, 

(2) identify problems that exist in the Navy’s current 
implementation of test and evaluation of aircraft 
weapon systems, 

(3) propose a reorganization of the numerous agencies and 
activities that conduct test and evaluation of Navy 
aircraft weapon systems. 

C . SCOPE 

In the broadest sense test and evaluation (T&E) takes 
place throughout a system’s life cycle (Pig. 1). In the 
Conceptual Phase test and evaluation takes the form of test 
planning and to a limited extent, simulation, breadboard, and 
laboratory testing. It isn’t until the latter portions of 
the Design Phase that test and evaluation takes the more 
commonly thought of form of the testing of prototypes, 
models, and subsystems. It is during the Design Phase and 
the Production and Installation Phase that test and evalua- 
tion activity reaches a peak, and it is this T&E that will 
be the area of concern of this thesis. T&E does not 
terminate after the Production and Installation Phase, but 
continues until the system is retired. The T&E that occurs 
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during the last two phases concentrates on determining 
optimal tactics to employ the system and testing and evalu- 
ating changes that are made to the system. These phases 
have minimal direct impact on the acquisition process and 
will not be an area of major consideration in this thesis. 

Although many problems exist with the test and evaluation 
of Navy aircraft, ships, and weapons, this thesis deals 
solely with the Navy's test and evaluation of aircraft 
weapon systems.^" Both the Navy and the contractor conduct 
T&E on each aircraft weapon system. This thesis will address 
primarily the T&E conducted by the Navy. 

D. METHODS OF RESEARCH AND ANALYSIS 

A thorough study of the available literature in the 
field of the management and organization of test and evalu- 
ation was conducted. This was followed by telephone inter- 
views with key personnel directly involved with Naval 
aircraft weapon system test and evaluation. 

The Annual Symposium of the Society of Experimental Test 
Pilots was attended. An entire major session was devoted to 
operational test and evaluation. This symposium afforded 
the opportunity to exchange ideas in the subject area with 
numerous key personnel involved in Navy, Air Force, Army, 
NASA, and contractor aircraft systems test and evaluation. 



^"Aircraft weapon systems is used herein to refer to the 
total aircraft including the weapon delivery system and the 
weapon itself. 
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After completing this preliminary research effort, it 
became evident that a high degree of parochialism exists 
throughout the Navy’s T&E community, with most people 
holding views that greatly favored the organization to which 
they were presently attached. Since the subject of defi- 
ciencies in the Navy's T&E organizations is a popular one, 
most people in each T&E activity shared similar ideas in the 
subject area. This is perhaps a result of the tremendous 
amount of dialogue and the large interchange of ideas that 
has taken place in this subject area. 

Research was, therefore, confined to personal or tele- 
phone interviews with a limited number of personnel involved 
in each of the various phases of Navy T&E. Because of the 
parochialism that exists within each T&E organization, it 
was possible to obtain an accurate cross section of that 
organization’s opinions of deficiencies in Navy T&E by 
interviewing only a few of its people. 

Visits were made to the Offices of the Secretary of 
Defense, the Offices of the CNO, the Naval Air Systems 
Command Headquarters, the Naval Air Test Center, and the 
Headquarters^ of the Operational Test and Evaluation Force. 
Appendix A contains a tabulation of the specific organiza- 
tions with which interviews were conducted. 

Fifty-seven people were interviewed either in person or 
by telephone. Interviews varied in length from fifteen 
minutes to three and one-half hours, and telephone conversa- 
tions varied in length from ten minutes to one and one-half 
hours . 
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II. BACKGROUND 



A. WHAT IS TEST AND EVALUATION? 

In order to ensure a common base for understanding the 
contents of this thesis, it is necessary to establish some 
fundamental definitions. A simple, yet all-inclusive, 
definition of test and evaluation which is utilized herein 
is the operation of a system or one or more of its component 
in a manner as similar as possible to its intended use and 
in an environment which approximates as closely as possible 
its operating environment for the purpose of gaining informa 
tion to evaluate the performance of that system or component 
The information obtained may be used to give a better under- 
standing of the system's capabilities, to identify short- 
comings, to develop improvements, to assist in designing 
the system's replacements or complements, to learn how to 
support and maintain the system, or for developing require- 
ments for training personnel in the operation of the system. 

Aircraft weapon system T&E can be divided into basically 
two types of testing: contractor T&E and military T&E. 

Contractor T&E is conducted by the contractor as a necessary 
step in his design and development of a weapon system and 
also as part of the contract requirements specified by DOD. 
Military T&E is that T&E conducted by military personnel to 
verify that the system meets the contract requirements and 
to ensure that the system being acquired is operationally 
suitable . 
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Aircraft weapon system T&E can also be divided into two 
different categories: Development Test and Evaluation (DT&E) 

and Operational Test and Evaluation (OT&E) . Comprehensive 
definitions of DT&E and OT&E which are in concurrence with 
the most recent DOD Directives are given in Appendix B. 

Briefly, DT&E is that T&E conducted by both the contractor 
and the military under the auspices of the developing agency 
for the purpose of demonstrating that the engineering design 
and development are complete, demonstrating that the system 
will meet specifications, determining whether the design 
risks have been minimized with respect to system producibility , 
and estimating the system’s military worth. 

OT&E is that T&E conducted by the military to estimate a 
system’s prospective military worth, operational effective- 
ness,^" operational suitability (including compatibility, 
interoperability, reliability, maintainability, and logistic 
and training requirements), and the need for any design modi- 
fications. It is also used to determine optimum weapon 
system employment tactics and force structure. OT&E should 
be conducted by operational personnel with qualifications 
similar to those expected to ultimately operate the system, 
and the tests should be conducted in as realistic an opera- 
tional environment as possible. 

In summary, DT&E can be divided into two types: 
contractor conducted DT&E and military conducted DT&E. 



^Operational Effectiveness is a measure of how well a 
system meets its defined operational requirements in its 
Intended operating environment. 
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OT&E can also be divided into two types: OT&E conducted to 

estimate a system's operational potential and OT&E conducted 
to determine how to best utilize the system. As a result 
of having received the most criticism, OT&E is currently the 
type of T&E that is being given major attention. The under- 
lying thesis is that the validity of quantitative test data 
improves as realism (testing in an operational/combat environ 
ment) is added to the environment of the test program. The 
achievement of this objective depends upon one's interpreta- 
tion of realism. It is easy to go overboard in the quest for 
realism to the extent of actually reducing the efficiency of 
the test program and adding needlessly to the cost of the 
test program. The current emphasis on OT&E appears motivated 
by an overzealous quest for:- the achievement of realism in 
the test environment. 

B. TEST AND EVALUATION CRITICIZED 

One of the earliest criticisms of T&E in DOD was made by 
the Military Aircraft Panel of the President's Scientific 
Advisory Committee (PSAC) in August of 1969. This panel 
recommended to the Secretary of Defense that an Operational 
Test and Evaluation (OT&E) organization be established in 
the office of the Secretary of Defense. It further went on 
to say that our lack of preparedness in previous wars could 
have been exposed if an adequate program for test and evalu- 
ation had existed [Ref. 4 ]. 

During this same time frame the Defense Science Board, 
the Joint Chiefs of Staff, the Bureau of the Budget, the 
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Comptroller General of the United States, and various 
Congressional Committees recognized that OT&E had been much 
less effective than necessary. Since OT&E is the fianl step 
in the research, development, test and evaluation (RDT&E) 
process, there appeared to be a potentially large pay-off 
for performing OT&E more effectively. 

Perhaps, the most noteworthy of the T&E studies was 



conducted by the President’s Blue Ribbon Defense Panel 
[Ref. 4], Citing the aforementioned criticisms of T&E and 



specifically OT&E, one Task Group of the Blue Ribbon Defense 
Panel performed an assessment of how OT&E was being conducted. 



It is interesting to note that this study, which became a 
major driving force behind changes to the overall DOD T&E 



process, looked only at OT&E rather than the entire DOD 
T&E effort. 



The main conclusions were: 



(1) OT&E can and should be planned and executed much more 
effectively than it has been in recent years. 

(2) OT&E results have not been made available to or used 
by DOD agencies which need them. 

(3) There is a requirement for an OT&E organization at 
higher-than-service level. 



(4) OT&E within the services is done most effectively when 
OT&E Organizations report directly to the Chief of 
Service, representing both the developer and user, 

but organizationally Independent of both . 

(5) Inadequate funding has adversely affected the conduct 
of needed OT&E. 



The panel went on to recommend: 

(1) An operational T&E Group should be established with 
civilian leadership reporting directly to the Deputy 
Secretary of Defense. 
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(2) A Defense Test Agency should be established with broad 
authority and responsibility for DOD test activities 
giving particular emphasis to OT&E. 

(3) The Secretary of Defense should communicate to the 
Military Departments the importance he assigns to OT&E 
and his conviction that OT&E is best served when 
independent OT&E organizations report directly to the 
Service Chiefs, Service Secretaries, or both. 

The Navy fared best of all services in this study, since 
it had in existence an independent OT&E agency, its Opera- 
tional Test and Evaluation Force (OPTEVFOR). Even though 
the Blue Ribbon Defense Panel demonstrated by many of its 
statements a misunderstanding of how the Navy conducted its 
T&E (see page 46), their recommendations received almost 
universal acceptance. 

In 1971 the Government Accounting Office (GAO) responded 
to Congressional requests and conducted a study of T&E for 
major weapon systems [Ref. 5]. Its main conclusions were: 

(1) The practices used in establishing test objectives 
were generally inadequate. 

(2) Most weapon systems reviewed did not have adequate 
plans for testing. 

(3) T&E for most weapon systems was not accomplished in 
a timely manner. 

(4) Complete and valid T&E data was not available to 
decision-makers prior to key decision points in the 
acquisition cycle. 

The GAO report recommended: 

(1) Completion of appropriate testing and evaluation prior 
to key decision points in the acquisition cycle. 

(2) Stringent controls over the granting of any waivers 
from required T&E. 

(3) Preparation of succinct summary reports by the testing 
agency for all levels of management. Interested 
management levels may wish to comment on these summary 
reports, however, they should not be permitted to 
change the basic content. 
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The Commission on Government Procurement in 1972 
echoed many of the previously cited criticisms [Ref. 2] of 
DOD T&E. Its emphasis was on the need to establish an 
operational test agency that is independent of the develop- 
ment and user organizations. 

The main criticisms coming from these studies appear to 
have been the lack of adequate "operational 1 ' test and 
evaluation and the lack of testing "independent" of the 
weapon system developers. Consequently, the words "operational" 
and "independent" are now used in every directive or instruc- 
tion that addresses T&E. 



C. NEW HIGH LEVEL T&E POLICY 

DOD Directive 5000.1 of July 13, 1971 established a new 
policy for the acquisition of major defense systems. In an 
attempt to correct deficiencies in DOD T&E cited in the 
previously discussed studies, DOD Directive 5000.1 states 
the following regarding T&E: 

"Test and evaluation shall commence as early as 
possible. A determination of operational suitability, 
including logistic support requirements, will be made 
prior to large-scale production commitments, making 
use of the most realistic test environment possible 
and the -.best representation of the future operational 
system available. The results of this operational 
testing will be evaluated and presented to the DSARC 
at the time of the production decision" [Ref. 1]. 

Memorandum issued by Deputy Secretary of Defense David 
Packard provided interim guidance to the services for imple- 
menting changes to their respective T&E systems until the 
issuance of a specific directive addressing T&E (Directive 
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5000.3 was issued in January 1972). This comprehensive 
directive [Ref. 6] covers all phases of DOD test and evalu- 
ation and is summarized as follows: 

(1) Development test and evaluation is to be planned, 
conducted, and monitored by the developing agency 
of the DOD Components.^ - 

(2) In each DOD Component there will be one major field 
agency separate and distinct from the developing/ 
procuring command which will be .responsible for 
operational test and evaluation and which will 
report results of its independent T&E directly to 
the Military Service Chief or Defense Agency 
Director. 

(3) Operational testing should be separate from develop- 
ment testing; however, combined testing may be 
conducted where separation would cause delay 
involving unacceptable risk or unacceptable increased 
cost . 

(4) At least an initial phase of operational test and 
evaluation (IOT&E) will be accomplished prior to the 
first major production decision sufficient to provide 
a valid estimate of expected system operational 
effectiveness and suitability. 



^DOD Component is a term referring to each of the main 
organizations that comprise the Department of Defense: Army, 

Navy, Air Force, and the Defense Supply Agency, for example. 
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(5) The Development Concept Paper (DCP) prepared for use 
at the time of the Program Initiation Decision for a 
major Defense System will identify the. critical 
questions and areas of risk to be resolved by test 
and evaluation. 

(6) When the DOD component proposes to initiate Full- 
Scale Development, it must present to the Defense 
Systems Acquisition Review Council (DSARC) the results 
of T&E accomplished to that date, an updated state- 
ment of critical questions and areas of risk still 
needing test to be resolved, and a detailed statement 
of test plans and milestones. 

(7) Prior to the first major production decision DSARC 
will assess the adequacy of test results to support 
a production decision and the adequacy of plans and 
schedules for any remaining testing. 

(8) The duties of the Deputy Director Defense Research 
and Engineering, Test and Evaluation (DD(T&E)) are 
delineated. He has across-the-board responsibility 
for the Office of the Secretary of Defense (OSD) in 
test and evaluation matters. 

One of the strongest motivating forces behind the changes 
that have taken place in DOD's test and evaluation process 
has been Congress. The report of the Senate Armed Services 
Committee from the 1st session of the Ninety-Second Congress 
[Ref. 7] fully supported the findings of the Blue Ribbon 
Defense Panel. The Committee reiterated weaknesses in DOD's 
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operational test and evaluation — lack of testing independent 
of the weapon system developer, service opposition to inde- 
pendent operational test and evaluation, lack of funds and 
facilities, and lack of high-level attention and management. 
The committee went on to state that insufficient OT&E in the 
past had resulted in the production and deployment of some 
weapon systems which were too complex to be effective. The 
committee reasoned that the military was misled into thinking 
its personnel would be able to maintain the systems satis- 
factorily, but the level of technical ability required to do 
so often exceeded that possessed in sufficient quantity by 
the military. The committee reasoned that adequate OT&E 
would have revealed this soon enough to make appropriate 
changes . 

In the same session of Congress, an amendment to the 
1972 procurement authorization bill (Public Law 92-156) was 
passed [Ref. 8]. This amendment. Section 506, requires that 
additional cost, schedule, and performance data be reported 
to Congress in support of the annual budget request and 
periodically throughout the year before major system procure- 
ment awards. are made [Ref. 9]. The required reports to 
Congress, called Congressional Data Sheets, must include 
data on OT&E results for each weapon system each year until 
procurement is completed. If OT&E has not been conducted, 
a statement of the reasons therefore and the results of such 
other T&E as has been conducted must be included in the 
report to Congress. 
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The Fiscal Year (FY) 73 budget request was accompanied 
by the newly required supporting program data reports. 

While similar supporting data were provided to Congress 
previously, PL 92-156 Section 506 standardized this procedure 
and made compliance mandatory. In passing this amendment, 
the intent of Congress appeared to be to provide a means 
of obtaining a broader range of data as a basis to make 
informed judgments on the merits of the programs and their 
funding requirements [Ref. 9]. However, these Congressional 
Data Sheets seem to be somewhat redundant of the information 
contained in the Selected Acquisition Reports (SAR’s) which 
are also required by Congress with copies sent to GAO. The 
SAR's have often been late in submission. For this and 
other reasons SAR’s have proven to be ineffective documents, 
but no current move is underway to abolish the SAR. Although 
Congress has not changed its methods of acting on the Armed 
Service Authorizations or Appropriations Bills since re- 
ceiving the Congressional Data Sheets, there is confidence 
in the Congress and the GAO that improved communications 
on weapon system development is being created between the 
Congress and DOD. 
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III. NAVY IMPLEMENTATION OF T&E POLICY 



A . GENERAL 

As stated in Section IIB, the Navy was the only service 
which had in existence at the time of the Blue Ribbon Panel 
Study an independent operational test and evaluation agency. 
This was the Operational Test and Evaluation Force (OPTEVFOR), 
which reported to the Chief of Naval Operations (CNO). It 
was a relatively small organization of approximately 1400 
officers and enlisted men and did not maintain its own 
test ranges or instrumentation. 

It was obvious that the most expedient method to comply 
with the DOD directives would the expansion of the role of 
OPTEVFOR to do the required OT&E. Appendix C contains a 
list of the various instructions and directives that pertain 
to Navy Aircraft Weapon System test and evaluation. The 
basic content with regards to T&E of a few of these Navy 
originated instructions will be discussed in order to show 
the Navy’s method of implementing the broad T&E guidelines 
given by Sec Def in DOD Directives 5000.1 and 5000.3. 

B. SECNAV INSTRUCTION 5000.1 [Ref 10] 

SECNAV Instruction 5000.1 is the Navy's implementing 
instruction of DOD Directive 5000.1. It assigns Commander 
Operational Test and Evaluation Force (COMOPTEVFOR) as the 
Navy's independent test agency for the required operational 
test and evaluation. The general sequence of test and 
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evaluation events under the new acquisition philosophy is 
given as follows: 

"1. Laboratory/contractor preliminary T&E of bread- 
board of demonstration hardware during the conceptual 
effort . 

2. Contractor/development activity T&E of subsystems 
and/or full-scale demonstrator hardware during full- 
scale development. 

3. Technical T&E conducted by the contractor with 
Navy participation during pre-production/production. 

4. IOT&E (Initial Operational Test and Evaluation) 

by or with the active participation of Navy operational 
forces prior to the major production decision. 

5. Navy OPEVAL (Operational Evaluation) prior to 
approval for service use and inventory acceptance 
(except for ships). 

6 . Navy follow-on T&E. 

7. Conduct of T&E by the Board of Inspection and 
Survey (BIS) and recommendation to CNO for service 
acceptability" [Ref. 10]. 

C. OPNAV INSTRUCTION 3960.8 [Ref. 11] 

OPNAV Instruction 3960.8, issued by CNO, is essentially 
the basic implementing instruction of DOD Directive 5000.3. 
OPNAV Instruction 3960.8 establishes guidance and policy 
for T&E of all Navy weapon systems, subsystems, components, 
and support systems. Specific areas discussed in this instruc- 
tion that are germane to this thesis are discussed below. 
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Development test and evaluation (DT&E) Is defined as T&E 
conducted under the sponsorship of the developing agency 
which Is undertaken to facilitate the evolution of a system. 
The primary objective of DT&E is given as the generation of 
essential and valid data upon which to base design decisions 
during the development process. 

Operational test and evaluation is defined as including 
all T&E efforts undertaken for the purpose of obtaining 
operational information throughout the system's life cycle. 
This T&E is used to support both the acquisition process 
and the optimum employment of the system. OT&E is broken 
into two phases: IOT&E (Initial Operational T&E) and.FOT&E 

(Follow-on Operational T&E ) ?- IOT&E is defined as that T&E 
accomplished by or under the supervision of the Navy's 
independent testing agency (OPTEVFOR) prior to the first 
major production decision. FOT&E is the continuing T&E of 
a weapon system conducted under fleet conditions by opera- 
tional personnel in order to verify system performance, 
validate correction of deficiencies previously identified, 
and refine tactical employment doctrines and requirements 
for personnel and training. 



^Follow-on Operational Test and Evaluation (FOT&E) is 
defined in OPNAV Instruction 3960.8 to include all OT&E 
following IOT&E. The useage of the term FOT&E has been 
rather limited and is not used in OPNAV Instruction 5^0. Z I7D 
or 3930.8 b, both dated after OPNAV Instruction 3960 . 8 . 
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The basic objectives of OPTEVFOR participation in T&E 
throughout a system's life cycle are given as: 

(1) Ensuring that early and realistic operational tests 
and evaluations are planned. 

(2) Ensuring that appropriate initial operational tests 
and evaluations are conducted prior to scheduled 
decision milestones, i.e. program initiation 
decision, full-scale development decision, and 
production decision 

(3) Providing an independent assessment of the operational 
effectiveness and suitability of the system to the 
CNO [Ref. 11]. 

The role of the Board of Inspection and Survey (BIS) 
for conducting acceptance trials on new ships and new model 
aircraft prior to Navy acceptance from the contractor is 

briefly discussed in this instruction. OPTEVFOR conducts 

\ 

OT&E to determine operational effectiveness and suitability, 
and BIS conducts T&E to determine acceptance for service 
use. It is recognized that these are closely related 
objectives, and this instruction directs that care be taken 
to preclude duplicate testing. One method outlined to 
accomplish this is the preparation of the Test and Evaluation 
Master Plan (TEMP) which is the responsibility of the 
Project Manager who works for the Chief of Naval Material. 

Figure 2, adopted from OPNAV Instruction 3960.8, shows 
the normal sequence of test events in the acquisition of 
an aircraft weapon system. 

The Director, RDT&E (OP-098) is assigned as the focal 
point for T&E policy and guidance for CNO. Under OP-098 
the Test and Evaluation Division, OP- 983 , is assigned the 
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NAVAL AIRCRAFT WEAPON SYSTEM T&E CYCLE 

FIGURE 2 



specific ressponsibillties of coordinating the overall Navy 
T&E effort. 

D. OPNAV INSTRUCTION 5440. 4 7D [Ref. 12] 

OPNAV Instruction 5440. 47D promulgates the mission and 
functions of OPTEVFOR. OPTEVFOR* s mission is to "opera- 
tionally test and evaluate specific weapon systems, ships, 
aircraft and equipments, including procedures and tactics, 
where required; and, when directed by CNO, assist development 
agencies in the accomplishment of necessary Development T&E" 
[Ref. 12]. 

The Director, RDT&E, (OP-098) is the OPNAV focal point 
for the Commander, OPTEVFOR. COMOPTEVFOR is, however, 

I 

authorized direct access to CNO and all subordinate offices 
as necessary to perform his assigned mission. 

E. SYNOPSIS OF NAVY IMPLEMENTATION 

The Navy had the simplest job of any of the Services in 
implementing changes to its T&E process so that it would 
conform to the new acquisition philosophy. Basically, the 
only organizational change made by the Navy was the creation 
of the relatively small (10 officers) OP- 983 , Test and 
Evaluation Division in the office of CNO’s Director of 
RDT&E. Figure 3 shows the present organizational structure 
for the major activities that are involved in aircraft 
weapon system T&E. Appendix D contains a summary of the 
functions and responsibilities of each of the major test 
activities . 
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PRESENT NAVY T&E STRUCTURE FOR AIRCRAFT WEAPON SYSTEMS 

FIGURE 3 



The role of OPTEVFOR has been greatly expanded and the 
roles of the other Navy T&E agencies have remained essen- 
tially unchanged by the Implementation of the .new T&E 
Instructions . 

The Program Managers In the S-3A and F-1*J programs 
faced the challenge of modifying their test programs to 
Include the new OT&E requirements. In both programs the 
leadership displayed by the program office in soliciting 
the necessary cooperation from the many T&E agencies was 
the primary ingredient which enabled the injection of the 
increased OPTEVFOR testing which had not been planned for 
previously. In the F-lk program the necessary cooperation 
was obtained through the formal establishment of the Joint 
Evaluation Team (JET) which was composed of test personnel 
from the Naval Air Test Center, Naval Missile Center, and 
Air Test and Evaluation Squadron Four (VX-^ who works for 
OPTEVFOR) [Ref. 131 . 



IV. ASPECTS OF CURRENT NAVY AIRCRAFT T&E POLICY 



Many deficiencies still exist in the Navy's T&E process, 
and others are being generated as a result of the new T&E 
policies. This section will discuss some of the deficiencies 
that exist in the Navy's T&E process as well as some of its 
enhancing characteristics which have not been changed. 

A. ORGANIZATIONAL DEFICIENCIES 

Because of the fragmented manner in which aircraft T&E 
is organized in the Navy, many problems result. These 
include : 

(1) The user is inadequately represented 

(2) Lack of independence of the T&E organizations 

( 3 ) Duplication of T&E responsibilities 

(4) Inadequate central management of NAVAIR T&E activities -+- 

(5) Adequate testing is not assured for non-major 
procurements 

(6) Requirement for independent T&E often ignores test 
agency expertise 

(7) No standardization in method of funding T&E agencies 

(8) Navy test range management is too fragmented. 

These deficiencies are discussed in the following sections. 

1 . The 'User is Inadequately Represented 

In each Navy procurement a dialogue that is vital 
for success is that between the user and the producer. Two 
such dialogues must exist: an external dialogue in which 

the Navy is the user, and the contractor is the producer, 
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and an internal dialogue in which the operating command is 
the user and the acquisition command is the producer. 

In the internal user-producer relationship the user 
is responsible for the operation and support of a system 
and is consequently also responsible for developing and 
stating the needs and concepts for the system. In other 
words the user provides the input requirements to which the 
producer designs the system. The producer is then responsi- 
ble for the design, development, production, and Installation 
of a system that meets the requirements of the user and 
that can be operated and supported cost-effectively. A 
constant dialogue must be maintained between the internal 
and external user and producer throughout the system's 
life cycle [Ref. 33* 

The external user-producer relationship is well 
defined and does not appear to present any significant prob- 
lems in regards to responsibility. Hov/ever, the internal 
user-producer relationship within the Navy is not so well 
defined. The Chief of Naval Material (CNM) represents the 
producer. The management of the development and production 
of a weapon system becomes the responsibility of one of 
CNM's System Commands and the specific responsibility of 
a Program Manager. The Office of the CNO (OPNAV) represents 
the user. Within OPNAV the Director, RDT&E (OP-098) initially 
becomes the Program Sponsor for the development of the 
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system, but he will eventually lose this sponsorhslp to the 
ultimate user sponsor. On aircraft programs it is the DCNO 
for AIR (0P-05) who will become the ultimate user sponsor, 
and he almost always assigns a Program Coordinator from 
his office to coordinate each program.^ - The OPNAV Program 
Sponsor is responsible for determining program objectives, 
time phasing and support requirements, and for appraising 
progress, readiness, and military worth of the weapon sys- 
tem. The OPNAV role as the representative of the user 
(i.e. the operating forces) appears well established up to 
the time of a favorable DSARC program initiation decision. 
Thereafter, OPNAV often becomes more of a proponent of the 
system being developed rather than an impartial representa- 
tive of the operating forces. This is especially true in 
today’s environment in which Congress is taking a more 
active role in each major weapon system’s development, and 
the CNO is compelled to frequently justify each Navy program 
to Congress. The opportunistic exploration that is often 
made by some members of Congress over any hint of a serious 
problem in a weapon system’s development adds further to 
OPNAV’ s serving only as a proponent of the system and to 
conceal any of its problems. 



1 The transition of Program Sponsorship from OP-O 98 to 
OP-05 should normally take effect upon the decision to 
procure the weapon system for service use. Of necessity 
there is a large period of dual interest and activity, 
but only one Program Sponsor has final responsibility. 
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OPTEVFOR, working for OP-098, is a test agency 
organizationally independent from the developing agency, 

CNM. Fortunately, OPTEVFOR is not put in a position like 
that of the Program Sponsor where it must conceal system 
deficiencies. OPTEVFOR is able to test and evaluate weapon 
systems while truly representing the user in determining 
if the weapon system is operationally suitable and whether 
it fulfills an operational need. 

OPTEVFOR, headed by a Rear Admiral, reports to 
CNO's Director of RDT&E, a Vice Admiral, who is often forced 
into a role where he must conceal weapon system development 
problems or risk losing the program at the Congressional 
budget hearings. Therefore, much of the valuable user 
dialogue that is generated by OPTEVFOR during the latter 
stages of a weapon system's development is lost through the 
OP-098 filter. 

The internal user-producer dialogue during the 
latter stages of a weapon system's development often becomes 
too producer oriented. The OPNAV program sponsor, forced 
into a system proponent role, becomes an ineffective 
representative of the user operational forces. 

The Navy's unilineal system, in which the operating 
forces and the Navy's acquisition forces report to the 
CNO, makes a clean separation between the internal user 
and producer forces extremely difficult to achieve. The 
creation of a bilineal system in which the Chief of Naval 
Material (heading the producer forces) and the Chief of Naval 
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Operations (heading the operating forces) report to a Navy 
Chief of Staff appears to be the only complete solution to 
this user-producer dilemma. However, much improvement can 
be made under the current unilineal system. It is beyond 
the scope of this thesis to discuss a change as drastic as 
would be the creation of a bilineal system. 

2 . Lack of Independence of the T&E Organizations 

The need for independent T&E has become a well agreed 
upon fact. As discussed in Section IIB much of the T&E 
conducted by the Navy has been criticized as being non- 
independent in that the .test agency worked for the developing 
agency. 

At the working level within the Navy's T&E structure 
there is little concern for lack of independence. No evi- 
dence indicates that T&E conducted by any test agency is 
biased and that the project team's reported results are not 
accurate results of tests conducted to the best of their 
ability. Navy program managers attest that the T&E conduc- 
ted by the NAVAIR T&E agencies is "independent", i.e. the 
P. M. is unable to influence the results. No one has really 
questioned the independence of the test personnel and 
their ability to conduct and report the actual test results 
they obtain. However, Congress has viewed the NAVAIR T&E 
agencies as not being independent, since the management 
wiring diagram shows them working for the developing agency. 
The accurate assumption is that NAVAIR has the power to 
completely disregard its DT&E agency test results, and the 
DT&E agency has no recourse. 






— - 
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A complete organizational separation is required 
to achieve the independence that is being sought by each of 
the critics of the Navy's T&E process. This is exactly what 
the Navy did in assigning the increased OT&E responsibility 
to OPTEVFOR, an agency that is organizationally separated 
from the aircraft developing agencies, NAVAIRSYSCOM and 
NAVMAT . 

However, OPTEVFOR is not a truly independent test 
organization. The Commander, OPTEVFOR reports to OP-09 8 , 
who is the program sponsor and was shown in the preceding 
section to play a producer-oriented system proponent role 
during OPTEVFOR' s involvement in the system's development. 
Although OPTEVFOR reports the results of its T&E directly 
to CNO, which is an acceptable procedure under DOD Directive 
5000.3, the OPTEVFOR reporting Chain-of-Command (COMOPTEVFOR ■> 
OP-O 98 -* OP-09 *»• CNO) is not within the intent of the current 
DOD philosophy. If COMOPTEVFOR must report to and through 
OP-O 98 , he cannot be considered legitimately independent of 
the developing agencies [Ref. 14], since he cannot convey the 
results of his T&E with enough force to effect a change in 
the system's development. 

An examination of OPTEVFOR 's funding and available 
physical resources shows further the inability of OPTEVFOR ^ 
to perform as a truly independent test agency. OPTEVFOR 
does not maintain complete fiscal control and management of 
all funding required for OT&E. In fact, all applicable costs 
expended for OT&E have never been separately identified. 



Without control of all the OT&E funding, it would appear 
that OPTEVFOR cannot independently plan and conduct OT&E. 

In any program whoever controls the funding controls the 
destiny of the program. 

Lack of control of the major portion of the physical 
resources required for OT&E (instrumentation, ranges, data 
processing centers, simulators, and test aircraft^) further 
reduces the independence of OPTEVFOR. A major portion of 
the physical resources utilized by OPTEVFOR is controlled 
by the developing agency. Yet, no formal, mechanism exists 
for the development of integrated plans for the utilization 
of these limited resources by OPTEVFOR and the DT&E agencies. 

3 . Duplication of T&E Responsibilities 

Much duplication of responsibilities exists among 
the major participants in the Navy's T&E process: the ' 

contractor, NAVAIR T&E agencies, OPTEVFOR, and Board of 
Inspection and Survey (BIS). The limited availability of 
personnel and physical T&E resources dictates that the Navy 
must eliminate needless duplication in order to achieve 
optimal utilization. Flight testing is the most expensive 
type of aircraft testing, and this is the area that will be 
given primary attention. 

The contractor is required to demonstrate satisfac- 
torily each segment of an aircraft's flight envelope 



1 Test aircraft are not permanently assigned to OPTEVFOR 
for IOT&E but only for follow-on T&E. 



(airspeed, altitude, normal accelerations, gross weights, 
etc.) before Navy test pilots are permitted to operate the 
aircraft within each flight regime [Ref. 15]. Although 
valuable in promoting safety and obtaining additional data 
under identical test conditions, this regulation causes many 
Navy test flights to be mere duplication of contractor test 
flights. 

Contractors are routinely required to redemonstrate 
compliance with the aircraft’s flight structural and perform- 
ance requirements at the Naval Air Test Center, Patuxent 
River, Maryland, after they have already performed the same 
tests at their own facility. This procedure is continually 
claimed by contractors to be a superfluous expense which 
could be eliminated by having the Navy witness these demon- 
strations at the contractor’s facility. The Naval Air Test 
Center has been an avid supporter of the need for this 
duplication of contractor tests for these reasons: 

(1) More credence can be placed upon results when tests 
are conducted at the Navy’s facility rather than the 
contractor ' s . 

(2) These limited demonstrations verify the contractor's 
instrument calibrations and put more credibility on 
all data collected at the contractor's facility. 

Compounding the contractor flight demonstration 
problem is the inability of the various Navy Plant Repre- 
sentative Offices (NAVPRO's) to witness competently all of 
the contractors' flight test demonstrations. This is the 
result of the limited flight test and aerodynamic expertise 
that is available at the NAVPRO’s. The various test agencies 



would, therefore, be required to send numerous qualified 
personnel to witness these types of demonstrations if they 
were conducted only at the contractor’s facility. 

The distinction between OT&E and DT&E implies two 
markedly different approaches to T&E which requires the need 
for separate test programs [Ref. 16]. Therefore, OT&E has 
been conducted primarily by OPTEVFOR and DT&E by the NAVAIR 
T&E agencies. Prior to DOD Directive 5000.1, there was a 
certain amount of duplication in the test programs of 
OPTEVFOR and the NAVAIR T&E agencies. This was unavoidable 
since completely different people were conducting tests on 
the same system, and a learning curve had to be developed 
by each of the respective testers. This same type of dupli- 
cation exists today and will continue to exist as long as 
separate agencies are performing tests on the same weapon 
systems . 

Before DOD Directive 5000.1, the tests were conducted 
in a sequential order and little overlap existed between the 
time-phasing of the tests conducted by the NAVAIR T&E agencies 
and OPTEVFOR. V/ 1th the current requirement of conducting 
Initial Operational Test and Evaluation (IOT&E) prior to the 
production decision, the amount of time overlap between 
OPTEVFOR and NAVAIR testing has greatly expanded. Figure k 
gives a presentation of the conduct of the various phases of 
T&E before DOD 5000.1 and today. 



The increased time overlap has increased the amount 
of duplication in testing. In the case of an aircraft both 
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OPTEVFOR and NAVAIR test agency flight crews (each agency 
needs more than one flight crew to assure corroboration of 
subjective evaluations) must conduct tests concurrently 
throughout the aircraft's evolution —prototype to production 
model — and the many iterations of design in between. 

Valuable flight test time must consequently be devoted to 
developing each flight crew's experience with each iteration 
of design in the aircraft prior to conducting meaningful 
tests . 



very similar testing objectives that exist for both BIS and 
OPTEVFOR. BIS, utilizing NAVAIR T&E agencies to perform the 
test and evaluation, is responsible to "discover and report 
deficiencies at the earliest possible time so that corrective 
action can be taken to ensure that aircraft delivered to 
operating units will be capable of carrying out the intended 
mission..." and to "recommend to the Secretary of the Navy 
the conditions of acceptance or rejection of a model aircraft 
for service use" [Ref. 17]. The majority of the pilots 
conducting the tests for BIS are graduates of the U. S. Navy 
Test Pilot School where they are taught to test for service 
suitability first and contract compliance second. 



sing the operational suitability of an item prior to the 
production decision and to "evaluate the operational effec- 
tiveness, suitability, and capability of tested weapons 
systems to meet stated needs and performance criteria" 

[Ref. 12]. 



The most obvious duplication in Navy T&E is the 




OPTEVFOR is responsible through its IOT&E for asses- 



The difference between BIS’s service suitability 
testing and OPTEVFOR's operational suitability testing is 
not apparent. The difference between BIS’s testing for 
specification compliance and OPTEVFOR’s testing against the 
stated needs and performance criteria is also not apparent. 
With the testing objectives being so similar, one can only 
expect duplication In the testing. Confusion often exists 
as to what testing is OT&E and what is DT&E, which tends to 
conceal much of the duplication in T&E responsibility. This 
confusion is illustrated by a quote from the Blue Ribbon 
Defense Panel which erroneously thought BIS was conducting 
OT&E. "Before Navy aircraft and ships are approved for use 
by the operational forces, they are subjected to another 
kind of operational evaluation which is much closer to true 
OT&E. This is done by a Board of Inspection and Survey (BIS), 
which is composed of experienced officers with appropriate 
Navy backgrounds" [Ref. 4, p. 23]. 

In attacking the problem of duplication of T&E 
responsibilities, the objective should not be to eliminate 
all overlap or duplication. It should, however, be to ensure 
that where such overlap or duplication exists, it is visible, 
controlled, purposeful, and contributes to improving the 
overall T&E process. 

4. Inadequate Central Management of MAVAIR T&E Activities 

The Naval Air Systems Command Test and Evaluation 
Coordinator was formally established in 1968. The Commander, 
Naval Air Test Center, Patuxent River, Maryland, was assigned 
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this additional duty, and was given a T&E Coordinator’s 
staff to assist in the performance of these duties which 
were separate and distinct from the other functions performed 
at Patuxent River. The creation of the T&E Coordinator 
stemmed from the recognition of the lack of a single focal 
point or reviewing authority for T&E in NAVAIR and the 
consequential lack of coordination among the NAVAIR T&E 
agencies [Ref. 18]. 

The NAVAIR T&E Coordinator's Charter [Ref. 19] — 
designates the T&E Coordinator as the focal point for all 
NAVAIRSYSCOM responsibilities in the coordination of T&E 
matters. It should be observed, however, that the T&E 
Coordinator is only a coordinator and has no line management 
authority over any of the NAVAIR T&E agencies, except NATC 
which he commands. The T&E Coordinator is responsible for 
assisting in the preparation of test planning in order to 
ensure optimum utilization of resources and minimum duplica- 
tion of tests. However, the T&E Coordinator only recommends 

V 



which agency should do the tests, and NAVAIR makes the final 
determination . 

The 'lack of a single management focal point for T&E 
within the internal NAVAIR organization continues to exist. 
Since NAVAIR has refused to delegate to the T&E Coordinator 
the authority to manage its T&E activities, many duplications 
of testing responsibilities exist and needless interagency 
conflict often exists. For example, NATC and NMC are 



assigned similar missions [Ref. 15> P* G-13]: 



NATC: "Coordinate and perforin test and evaluation 

of aircraft weapons systems, their components 
and related equipment, ..." 

NMC: "To perform test, evaluation, development support 

and exercise engineering cognizance as assigned 
of Naval weapons, weapon systems, and related 
devices." 

NMC has test responsibility for missiles and the 
interfaces with the various launch platforms (aircraft) 
while NATC has test responsibility for the aircraft and the 
interfaces with the various weapons that it will deliver. 
Because of the large overlap in responsibility, much conflict 
has arisen in the past between Weapon System Test (WST - a 
division of NATC) and NMC. Participants in the F-l^i/Phoenix 
Missile Test program have stated that more conflict existed 
between these two NAVAIR T&E agencies than between either 

t 

NATC or NMC and VX -4 who reports to OPTEVFOR. 

Another example of NAVAIR duplication of testing 

responsibilities exists in the area of aeromedicine . Below 

is a list of the agencies directly involved with the T&E 

of life support and aircrew protective equipment: 

Naval Aerospace Recovery Facility, El Centro, CA. - 

conducts DT&E on escape and recovery systems 
(see Appendix D) 

NATC - Aeromed Branch of Service Test Division 
NMC - Life Support and Life Sciences Department 

NADC, Warminster - Although not reporting directly to 

NAVAIR, NADC receives approximately 85 percent 
of its funding from NAVAIR and does extensive 
development and testing in the field of 
aeromedicine 

Each of these agencies is involved with the development and 
T&E in the aeromedical area. The need to have aeromedical 
capabilities and expertise at each of these facilities is 



questioned as needless duplication which is permitted to 
exist and expand because no central authority is controlling 
NAVAIR's T&E activities. 

5. Requirements for Independent T&E Often 

Ignore Test Agency Expertise 

LTGEN Alfred Starbird (USA-Ret) is the Deputy 
Director of Defense Research and Engineering, Test and 
Evaluation (DD(T&E)) in the Office of the Secretary of 
Defense (OSD) . As DD(T&E) he has become a main driving 
force in ensuring that each of the Services conduct T&E in 

1 

accordance with DOD Directive 5000.3. 

DD(T&E) is responsible for reporting to the DSARC 
"at each major milestone decision point his assessment as to 
the adequacy of the identified critical issues and questions 
to be resolved by test and evaluation, test plans and 
schedules, and the adequacy of the accomplished T&E to 
justify the action recommended for that milestone decision" 
[Ref. 6]. OPTEVFOR, as the Navy's independent operational 
T&E agency, is the main source of DD(T&E)'s T&E data. 

Although much of the development agency's T&E data are made 
available to DD(T&E), it is the OPTEVFOR evaluation that is 
required before DD(T&E) will be satisfied that adequate T&E 
has been accomplished to meet a milstone T&E objective. In 
the event of a conflict, the OPTEVFOR opinion will take 
precedence . 

As will be discussed in Section IV. B. 4, OPTEVFOR 
often lacks the technical expertise to conduct a particular 
test of a system or subsystem. Yet DD(T&E) has, in the past, 



required that OPTEVFOR conduct an evaluation before continu- 
ing with the development program, since DD(T&E) was not 
satisfied with only the results from an NATC (a developer's 
T&E agency) evaluation. In other words, OPTEVFOR is being 
tasked to conduct T&E on the basis of it s independence 
rather than on its testing capability. The result of such 
actions is that OPTEVFOR is being compelled to build up its 
technical expertise and expand its testing capabilities into 
areas where this capability and expertise exists in other 
test agencies. 

\ 

The distinction between OT&E and DT&E is not always 
apparent. Many T&E milestones for aircraft procurements 
could easily be shown to be met by conducting only so called 
development testing; e.g. meeting a performance requirement 
of maximum airspeed or altitude. However, as long as DD(T&E) 
requires the opinion of the independent test agency, OPTEVFOR, 
the Navy will be forced to increase the capability of 
OPTEVFOR. This will, in turn, further increase the duplica- 
tion of responsibilities between OPTEVFOR and the development 
test agencies. 

6 . Adequate Testing is Not Assured for Non-Major 

Procurements 

DOD Directive 5000.1 and the various implementing 
instructions pertain to major programs (RDT&E cost in excess 
of 50 million dollars or production cost in excess of 200 
million dollars). However, all the directives state that 
the principles are to applied to all programs. Since DSARC 
is only involved with major programs, there is no exogenous 
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system to ensure that adequate and independent T&E is 
conducted prior to making the advanced development or the 
production decision for non-major programs. 

The lack of any explicit policy for T&E of non-major 
systems has given considerable latitude to the Program 
Manager. Many projects are not subject to automatic project 
assignment by CNO, and consequently OPTEVFOR is often not 
cognizant of the existence of projects or their need for 
operational evaluation. OP-983 publishes a quarterly list 
of all assigned RDT&E projects to be prosecuted by OPTEVFOR, 
but no means exists for ensuring that every- project^ requ iring 
an operational evaluation is so assigned. What this means is 
that the Program Manager is able to conduct inadequate T&E, 
or he can even obtain T&E from a number of agencies until he 
receives the results he desires; and his actions may not even 
be criticized. 

Without questioning the integrity of the Navy's 
Program Managers, one can certainly speculate on the outcome 
of a situation where the Program Manager is typically facing 
a cost growth combined with a slippage in schedule but has 
no clearly established requirement to obtain an "independent" 
test and evaluation of his system. His likely course of 
action would be to obtain the least expensive (to his program) 
suitable T&E that is available . Within the current Navy T&E 
structure many avenues to conduct T&E are open to the Program 
Manager: a NAVAIR T&E agency, OPTEVFOR, or a fleet unit. 



51 



The following is an example which occurred in 1970 
in the development of the night carrier landing simulator 
for the A-7E airplane. The Program Manager had the fleet 
replacement A-7 training squadron (VA-122) conduct the Navy’s 
test and evaluation. The Naval Air Test Center and speci- 
fically the Carrier Suitability Branch, the technical experts 
for testing aircraft and associated systems for use in the 
carrier environment, did not participate in the test and 
evaluation of the night carrier landing simulator. The 
VA-122 T&E was certainly less expensive than would have been 
T&E conducted by a NAVAIR test agency. In addition the 
likelihood of uncovering deficiencies (costly to the Program 
Manager to correct) with the simulator was far less by having 
untrained testers conducting the evaluation than it would 
have been if the critical voice of NATC had been given the 
opportunity to speak. Although the A-7E night carrier 
landing simulator has proven to be a valuable system, the 
example does illustrate how the Program Manager has the 
latitude to bypass his own NAVAIR test agencies as well as 
OPTEVPOR. 

Another illustration of the Acquisition Manager’s 
(the Program Manager for lower cost programs) latitude in 
conducting T&E is the case of the HLU-196A/E Powered Bomb 
Hoist. The success of this hoist is essential for the 
implementation of the Improved Rearming Rate Program aboard 
the Navy's attack carriers. T&E of the Bomb Hoist was 
conducted by NATO and the Naval Weapons Evaluation Facility 
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(NWEF) [Ref. 20], NATC recommended against clearance of 
the HLU-I 96 A/E for use with all explosive ordnance and NWEF 
recommended restriction against use of the hoist with 
nuclear weapons until certain safety deficiencies were 
corrected. Yet, the feature article of the April 1973 
edition of Naval Aviation News [Ref. 21] reported that 
carrier testing of the HLU-I 96 A/E had been very successful 
and implied that NATC and NWEF had obtained similar results 
in their tests. Apparently the four-day HLU-196A/E evalua- 
tion conducted aboard USS Enterprise during combat operations 
by operational Navy ordnancemen showed the HLU-I 96 A/E to be 
very effective. The design deficiencies (high failure possi- 
bility of two critical springs) which NATC and NWEF considered 
to have such an adverse effect on safety never showed up in 
this "operational evaluation." 

It appears that the Acquisition Manager was unwilling 
to accept the negative T&E results from NATC and NWEF. Hope- 
fully, the serious design deficiencies in the HLU-196A/E can 
and will be corrected before a catastrophic accident occurs. 
One would like to believe that the acquisition manager has 
not considered his system's design complete, merely because 
someone was found to report his system as being a "significant 
step toward a mobile automatic weapons loader" [Ref. 21, 

P. 14]. However, the Navy's acquisition process for non-major 
procurement provides an inadequate check to protect against 
prematurely terminating the correction of deficiencies. 
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7. Methods of Funding 



As pointed out in Section IV. A. 2 and IV. A. 3, the 
costs for conducting T&E cannot be totally identified. 
Further compounding the situation is the fact that many 
different methods are used for funding the various T&E 
agencies. Some development T&E agencies are operated under 
Navy Industrial Funding where both direct and indirect costs 
are charged to each T&E project, and the funds are then 
transferred from the respective Program Manager. Other 
agencies utilize industrial funding where only direct 
charges are Incurred by the project, while still others 
never even charge military personnel working directly on 
the project as a direct cost to the project. Consequently, 
the cost to conduct T&E varies significantly from agency to 
agency. 

In the past, Program Managers have been forced by 
monetary constraints to have certain tests conducted by a 
T&E agency which was not as well suited to perform the T&E, 
but whose funding method resulted in much' lower cost for 
the Program Manager and his particular program. However, it 
is quite possible that the overall cost to the Navy might 
have actually been higher, but this is not determined or 
considered in the Program Manager's decision making process. 
Obviously a system which forces a Program Manager to make 
decisions based upon cost tradeoffs must reflect realistic 
costs which, when compared, enable the Program Manager to 
make decisions which give valid cost savings. 
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The funding for OPTEVFOR is different from that of 
the development T&E agencies in that more funding sources 
are involved. The VX Squadrons, the Air Test and Evaluation 
Squadrons that conduct aircraft T&E for COMOPTEVFOR, receive 
part of their funding from their respective Type Commanders 
(COMNAVAIRPAC or COMNAVAIRLANT) . Fleet Commanders also fund 
OPTEVFOR indirectly through their Operating and Maintenance 
(O&M) funds by providing fleet services and support for OT&E. 
The remainder of OPTEVFOR funding comes from the various 
Systems Commands to support OT&E on a project by project 
basis [Ref. 15 ]. As long as the funding for OT&E is not 
centrally managed, OT&E will not have fiscal accountability, 
and cost-effectiveness trade-offs for conducting different 
types of T&E will be nearly impossible to perform. 

8 . Navy Test Range Management 

Appendix E contains a list of the major Navy ranges 
and physical test facilities that are utilized in aircraft 
weapon system T&E. The ranges are controlled by a number of 
different commands. The only central range authority is the 
Naval Material Command (NAVMAT) Navy Range Manager, who only 
has line management control over the NAVMAT and NAVAIRSYSCOM 
controlled ranges. It is significant to note that OPTEVFOR 
does not have control over any range or physical test 
facility. Consequently, OT&E is conducted at ranges and 
facilities which are primarily development T&E oriented. 

The tests are planned and conducted around the specific range 
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or facility requirements, rather than on the basis of what 
is required for optimal OT&E. 

The cost to operate the ranges and physical facilities 
cannot be isolated, since they are buried in the budgets of 
the various controlling agencies. Without central range 
management or knowledge of the overall operating costs, it 
is a near impossible task to justify the various ranges and 
make rational decisions when forced to reduce range assets. 
Since the ranges available for military use determine the 
location and the functions of the various T&E facilities, 
this area is critical to T&E. During this period of increased 
civilian interest in the vast amount of real estate controlled 
by the military, it behooves the Navy to devote its best 
efforts to ensure that adequate ranges and facilities will 
exist for T&E in the future. 

The under-utilization of many test ranges makes them 
candidates for elimination or conversion to some other use. 

The General Services Administration is conducting a continual 
survey to identify under-utilized and/or poorly utilized real 
property holdings of all Government agencies [Ref. 22]. The 
Navy must determine its total Navy Range requirements, and 
then determine if it has an excess or a shortage. With so 
many Navy agencies controlling ranges and facilities, it 
becomes a difficult task to separate the actual Navy range 
requirements from the internally generated requirements where 
each agency tries to "keep what it has." 
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B. OTHER DEFICIENCIES 



Besides the organizational features of the Navy's T&E 
process many other phenomena exist which cause the Navy's 
T&E process to be deficient. This section will discuss 
these other deficiencies: 

(1) Overlap between DT&E and OT&E 

(2) Inadequate Test Planning 

(3) Inadequate operational test objectives 

(4) Inadequate technical expertise in OPTEVFOR 

(5) Lack of a Weapon System Test and Evaluation School 

(6) Late test results 

(7) BIS scheduled prior to DSARC III 

(8) Shortage of analysts 

(9) T&E not always treated as a continuing process 
1 . Overlap Between OT&E and DT&E 

Perhaps the greatest problem facing military T&E 
today is the lack of a uniform understanding of the differ- 
ences between OT&E and DT&E. The definition of OT&E and 
DT&E presented in Appendix B may seem quite explicit to the 
uninitiated. The consensus throughout the Navy's aircraft 
T&E community, however, is that the major portion of T&E 
conducted by the military falls somewhere in _a "grey area" 
that is not purely DT&E nor purely OT&E. 

Since the majority of T&E, therefore, is conducted 
in an area that is not clearly the responsibility of the 
DT&E agency or the OT&E agency, parochial attitudes are 
typical and often hinder cooperative testing. Each agency 
attempts to lay claim to the larger or certainly the more 
desirable portions of the system's particular testing 
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equirements. There exists a deflnate tendency for opera- 
ional test pilots to want to become more Involved with the 
ore glamorous experimental or developmental testing, 
ecently, more and more pilots with only VX/operatlonal T&E 
ackgrounds have been accepted for membership In the Society 
f Experimental Test Pilots, an organization which requires 
^tensive experience In developmental or experimental flight 
Esting for full membership. These VX pilots have acquired 
ie requisite experimental and developmental test pilot 
xperience, but they did it while in operational test pilot 
Lllets . 

A similar tendency exists on the part of the develop- 
snt test agencies. The development test pilot/NFO (Naval 
light Officer) resents being put into the current situation 
Here he feels that he is asked only for technical opinions, 
id OPTEVFOR is relied upon for operational opinions. The 
svelopmental T&E project officers have operational back- 
rounds comparable with those of the operational T&E project 
'ficers in OPTEVFOR. The only significant difference is 
lat the DT&E project officers spend a year in Test Pilot 
'hool after leaving their last operational billet and prior 
being assigned to a DT&E agency. Consequently, the DT&E 
'oject officers adamantly refuse to accept roles that are 
olated from operational suitability, and they tailor their 
• sts to look at the operational aspects as well as the 
-chnical aspects of a weapons system. 
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This cross-pollination of mission roles between the 
DT&E agencies and OPTEVFOR probably causes each side to 
become better "military T&E" organizations. However, the 
U. S. Navy cannot afford the luxury of developing this 
expensive duplication. As long as separate agencies are 
being tasked to perform T&E which does not lend itself to 
an easy separation into DT&E and OT&E, parochialism will 
continue to grow at the expense of T&E efficiency in terms 
of cost and performance . 

2 . Inadequate Test Planning 

DOD Directives 5000.1 and 5000.3 require early 
planning for T&E. Prior to the DSARC I Program Initiation 
decision the critical questions and areas of risk that have 
to be resolved by T&E must be identified. At this time the 
test objectives, schedules, and milestones must also be 
determined. This can only be accomplished through extensive 
early test planning with the assistance of the T&E agencies. 

Unfortunately, the_T&E agencies are not-involved in 
the early stages of test planning. A Test and Evaluation 
Master Plan (TEMP) is required to be prepared as early as 
possible in the acquisition cycle. It is a management docu- 
ment which describes how and when developmental and opera- 
tional test objectives will be met, and its preparation is 
the responsibility of the development agency's Program 
Manager [Ref. 11]. The T&E agencies become involved only 
through a review and comment process rather than being 
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ln t imately involved in determining Jtest objectives and how 

best to determine if they are met. 

<n 

To translate this situation into the commercial 
DOD market place, when DOD procures something for a contrac- 
tor, the user, in this case DOD, tells the contractor or 
producer what tests will be performed to demonstrate the 
acceptability of the particular item. The contractor may 
assist in the structuring of the tests or in recommending 
substitute tests, but the responsibility for test planning 
to determine service suitability clearly must remain with 
DOD. In going back to the Navy's internal situation, a 
similar situation should but does not exist. The producer, 
in this case the developing agency, has the responsibility 
of preparing the test plan for each particular system. The 
user, in this case OPNAY, merely assumes a review and comment 
role . 

Certainly the Program Manager must retain the 
responsibility for test scheduling and for satisfactorily 
meeting the test objectives. However, the user and the 
T&E experts should have the primary responsibility for the 
determination of the test objectives and how, when, where, 
and by whom they can best be tested to determine if they are 
being met satisfactorily. 

3 . Inadequate Operational Test Objectives 

The preceding section discussed the inadequacy of 
test planning in the early stages of a system's development. 
This has a most adverse effect on OT&E. Very frequently 
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OPTEVFOR is tasked to perform an "operational evaluation" 
on a system or subsystem, but a standard against which to 

test does not exist. Unfortunately, the operational require- 

I >-* • ' 

ments for many weapon systems are never well promulgated nor 
explicitly defined. 

The Specific Operational Requirement (SOR) is the 
requirement document from CNO to which a system is designed. 
The SOR is supposed to define the required weapon system 
capability in terms of mission requirements, operational 
concept, and performance constraints [Ref. 23]. It would 
appear that the SOR would be the "standard" against which 
OPTEVFOR could design its tests. In reality, the SOR is 
often incomplete or outdated by the time the system is ready 
for any T&E, or the SOR has not been written and approved 
by the time T&E has commenced. One further deficiency is 
that the SOR usually contains little in the way of test 
planning or test objectives — that is left entirely to the 
developing agency in its response to the SOR, the Technical 
Development Plan (TDP) . 

The TDP is the developing agency's management docu- 
mentation of its proposed actions and procedures, and the 
resources which are required in order to achieve the capa- 
bility described in the SOR. The TDP is a primary planning 
document for the development, production, installation, 
integrated logistic support, reliability and maintainability, 
personnel training, and test and evaluation of the weapon 
system [Ref. 23]. The in-depth analysis that goes into the 
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TDP translates the requirements of the SOR (If one exists) 

Into specific performance and development objectives 
against which DT&E agencies can test. However, the TDP Is 
not the document against which OPTEVFOR should be testing. 

A comprehensive analysis should be conducted early 
in the requirements determination phase (Conceptual Phase) 
to develop operational effectiveness criteria. These cri- 
teria would serve the valuable function for establishing 
a general method to evaluate and assess whether the system's 
development is meeting the operational needs. Operational 
effectiveness criteria should facilitate the writing of a 
flexible test plan which is required by OT&E. The SOR or a 
suitable replacement document should contain such operational 
effectiveness criteria. This is clearly the role of the user, 
i.e., OPNAV, and it should not be delegated to the developing 
agency . 

4 . Inadequate Technical Expertise in OPTEVFOR 

There appears to be a common belief that the main 
requirement for an OT&E project officer is that he possess 
current fleet operational experience. While extensive 
operational experience is a necessity for the molding of a 
competent OT&E project officer, his training should also 
include schooling in the technical disciplines th at are 
involved in his project; aeronautics, electronics , physics, 
and test methods (including stat istics and test d esign) . 

It is a well established requirement that the 
majority of the project officers (pilots and Naval Flight 
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Officers) involved with DT&E are graduates of a formal Test 
Pilot School where the discipline of T&E is taught as are 
also many basic technical fundamentals. In addition each 
DT&E agency has an extensive civilian technical work force 
which is composed of aeronautical, mechanical, and electrical 
engineers, mathematicians, physicists, and computer special- 
ists. An in-depth technical background has been recognized 
by the DT&E agencies as an absolute requirement for con- 
ducting T&E of a sophisticated weapon system. 

Many OPTEVFOR project officers believe that opera- 
tional test objectivity is decreased as technical expertise 
is increased. However, the increased T&E role that OPTEVFOR 
is being forced to fulfill is causing OPTEVFOR to recognize 
the need to increase its level of technical expertise. The 
number of project officers possessing both operational 
experience and technical capability must be expanded if 
OPTEVFOR is going to fulfill adequately its T&E objectives 
while efficiently utilizing test resources. 

By designing efficient test plans, millions of 
dollars can often be saved, while obtaining even more 
conclusive results. The design of such test plans cannot 
be accomplished by personnel possessing only a primary 
attribute of recent operational experience. Operational 
testing should be conducted by personnel having similar 
qualifications to those who will ultimately utilize the 
system, but this should not preclude personnel from possessing 
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the requisite technical capabilities from designing and 
planning the tests and analyzing and evaluating the results. 

OPTEVFOR recently evaluated a system in USS Ranger. 

The system's performance was believed to be affected by 
latitude, longitude, look angle, and other parameters. 

Testing for latitude and longitude effects would necessitate 
relocating the test force all over the world. Fortunately 
a project officer assigned to this program had an excellent 
technical background. He was able to show by some relatively 
simple preliminary tests that both latitude and longitude 
had an insignificant effect on one performance parameter in 
question. Latitude and longitude would affect this parameter 
only as a result of the consequential changes in look angle. 
This knowledge could provide considerable cost savings, since 
only look angle must be varied to complete this portion of 
the test. One can only wonder how many tests consist of 
expensive testing for the effects of parameters which could 
have been shown to be insignificant by a skilled analyst. 

Another area where OPTEVFOR’ s technical expertise is 
lacking is in the testing of computer programs (software), 
which have become an integral part of every current weapon 
system. The results of inadequately designed tests of these 
weapon systems will not show where system problems exist, 
i.e. in the computer program, in the hardware, in the operating 
procedures, etc. If one is going to expend the effort and cost 
to conduct the tests, it is only logical to expect that the 
results will be able to show where problems exist, not merely 
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that the system operation is good or bad. In order to 
design adequate operational or developmental tests for 
computer software, an In-depth knowledge of the system 
operation is required, as is also a good basic understanding 
of the technical principles involved. 



obtaining more civilian analysts and other technical special- 
ists and by increasing the formal training given to the 
project officers. The assignment of only one civilian, an 
analyst, to each OPTEVFOR VX squadron is an inadequate core 
of technical expertise around which the VX squadron can 
perform efficiently and effectively. The recent efforts by 
OPTEVFOR to obtain more officers with graduate technical 
degrees from the Naval Postgraduate School is a move in the 
right direction, but much more emphasis has to be given to 
this area. 

5 . Lack of a Weapon System T&E School 



before project officers are assigned to an OPTEVFOR project. 
VX-4 project officers are probably the most formally trained 
in OPTEVFOR. They usually attend a two week training 
program at Deputy COMOPTEVFOR, North Island, California and 
then a formal three day training course on the operation and 
utilization of the Pacific Missile Range (PMR), their primary 
test site. But the main part of every OPTEVFOR project 
officer's training is "on-the-job training" — also called 
"learning from one's own mistakes." In the T&E environment 




Technical expertise in OPTEVFOR can be increased by 



Very little formal training is required or is given 
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mistakes are too costly a teaching tool. It costs approxi- 
mately $36,000 per hour to utilize the PMR. This should 
compel any user to ensure that his tests are conducted as 
efficiently and expeditiously as possible with minimal 
likelihood of having to repeat. 

Although the majority of DT&E agency project officers 
have attended a formal one year course of study at a Test 
Pilot School, the knowledge they receive is primarily in the 
discipline of aircraft flying qualities and performance T&E. 
While this is essential to anyone becoming a project officer 
in the Flight Test Division of NATO, approximately two-thirds 
of the U. S. Navy Test Pilot School graduates go to project 
officer jobs where there is very little need to utilize any 
of the flying quality and performance test techniques. What 
is lacking in the training of these project officers is 
extensive training in the T&E of complex weapons systems. 

The U. S. Navy Test Pilot School has introduced a course 
entitled Integrated Weapons Systems, but it composes only 
7 percent of the formal classroom instruction time and serves 
the purpose of giving the student only an introduction into 
the complex field of weapon system T&E (as opposed to aircraft 
vehicle T&E). 

As with OPTEVFOR "on-the-job training" then becomes 
the primary training method for the DT&E project officer who 
becomes involved with weapon system testing. But the DT&E 
project officer has many distinct advantages over his OPTEVFOR 
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counterpart. The Test Pilot School has given the DT&E 
project officer considerable knowledge and experience in 
the field of T&E (i.e., how to plan, conduct, and analyze 
tests efficiently), and it has also taught him how to write 
concise, accurate test reports. 

DT&E project officers receive more formal training 
than their OPTEVFOR counterparts. However, a significant 
deficiency exists in the training of _both- -types ...of project 
officers. Nowhere are they taught how to test a complex 

( 

weapon system, how to separate computer software problems 
from system hardware problems, how to design an optimal test 
using a statistical design of experiments approach, etc. 

Not only do the project officers receive inadequate 
formal training, but so do the civilian engineers throughout 
the T&E community. Most of the civilian engineers working 
in the Flight Test Division of NATC complete the entire 
academic curriculum of the Navy Test Pilot School. However, 
formal training in T&E does not exist for civilian engineers 

i 

at other T&E agencies. Although the civilians possess 
technical or engineering degrees, very few civilians 
reporting to a T&E agency have had any prior experience or 
training in the discipline of T&E. Consequently, costly 
on-the-job training in T&E is again the technique utilized to 
train the civilian engineers and analysts. 

6 . Late Test Results ' 

The new DOD T&E policies are intended to correct the 
deficiency of T&E being conducted too late to affect the 



67 



system’s development. However, Program Managers are still 
plagued with the problem of not getting test results soon 
enough from the test agencies which they have requested to 
conduct certain tests. This Is usually not a problem with 
NPE or BIS reports where the schedule for report submission 
Is very rigid, and all T&E agencies are forced to comply. 

The problem arises with the myriad of minor T&E projects 
that are conducted during every major weapon system’s 
development. The test reports from the DT&E agencies go 
through such a lengthy review process that they are frequently 
received by the Program Manager too late to assist in his 
decision making process. Figure 5 shows a typical DT&E agency 
test report review chain. 

Not every test report is required to follow the 
complete review chain, but all of the test reports dealing 
with important projects do follow it. Some type of formal 
review chain is an absolute requirement to ensure that the 
test agency’s main output, test and evaluation reports, do 
not fall below the established minimum level of quality, 
and to ensure that misleading statements or statements not 
supported by data are not made in the reports. However, 
when the review chain becomes so time consuming that it 
becomes the major obstacle in the conduct of T&E, a signi- 
ficant change should be made . 

7 . BIS Scheduled Prior to DSARC III 

OPNAVINST 3960.8 shows BIS commencing along with 
OPEVAL prior to the DSARC III major production decision. 
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Typical DT&E Agency Test Report Review Chain 

FIGURE 5 
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(See Figure 2). The function of the OPEVAL is to determine 
if the system is ready for production, and the function of 
BIS is to determine if the production aircraft are acceptable 
for service use and whether the system's contract specifica- 
tions have been met [Refs. 2 and 20]. 

If this instruction is to be followed in regard to 
the time sequencing of BIS and OPEVAL, the functions of BIS 
aircraft trials will have to be modified. However, if it is 
modified so as to fulfill a function prior to the major 
production decision, this function would obviously have to 
be related to determining whether a favorable production 
decision should be made. As discussed previously (Section 
IV. A. 3), the roles of BIS and OPTEVFOR are too similar at 
present, and such a change to the function and objectives 
of BIS would exacerbate the duplication that already exists 
between the function of BIS and the functions of OPEVAL. 

The other alternative is to have BIS remain as service 
acceptance trials and have it occur after OPEVAL, which is 
a radical change from the current sequence of aircraft T&E. 

;8. Shortage of Analysts 

The requirement to utilize statistical analysts for 
designing tests and analyzing the data for statistical 
significance has long been known and practiced by OPTEVFOR. 
The - DT&E agencies are still living in a world where they 
believe "everything" is deterministic, and very seldom is 
the statistical aspect of data fully analyzed. This is 
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typical of the entire scientific and engineering community. 
Only within the last five to ten years has emphasis been 
placed upon the probabilistic aspects of engineering. 

The DT&E agencies need to devote more attention to 
the investigation of the statistical aspects of their tests. 
A large amount of DT&E testing is done where statistics 
(i.e. means, variances, confidence levels) are calculated, 
but a statistical analyst is'seldom if ever used to verify 
the credibility of the test. While a large part of "our 
world" might be distributed normally,"'' in-many instances 
this assumption can lead to some significant errors in the 
conclusions one draws from data. 

Although OPTEVFOR has recognized the need for 
analysts, it still has serious T&E problems that can be 
traced to the lack of sufficient analysis expertise. There 
is presently one civilian analyst assigned to each VX 
squadron and four military and four civilian analysts (two 
are permanent Government Service employees) assigned to 
OPTEVFOR Headquarters staff [Ref. l4]. The problem with the 
analysts is twofold: (1) low experience level, and 

(2) inadequate numbers of analysts to perform all required 
functions . 



^"Populations of random variables are distributed according 
to different types of statistical distributions. The Gaussian 
or Normal distribution is one of the most common and a great 
many test parameters and conditions can accurately be 
described by the Normal distribution. However, normality 
should always be tested for and not merely assumed. 

Incorrectly assuming a Normal distribution when some other 
distribution is appropriate could lead to erroneous conclusions 
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The civilian analysts assigned to the VX squadrons 
and two assigned to OPTEVFOR headquarters work for the 
Operations Evaluation Group (OEG), a part of the Center for 
Naval Analysis (CNA) . These OEG representatives are assigned 
to OPTEVFOR for a two year period (recently increased from 
only one year). For the OEG representative this two year 
period is his field experience tour. Generally, the OEG 
personnel are relatively junior when they are assigned to 
OPTEVFOR, and consequently a large amount of their time is 
spent in on-the-job training. The best career promotion 
opportunity for the OEG personnel appears to exist when they 
are assigned to jobs within CNA headquarters. Consequently, 
there is little motivation for an OEG representative to 
continue working for OPTEVFOR despite the fact that he may 
find his job challenging, rewarding, and essetnial. As a 
result of these circumstances, the experience level and 
caliber of the OEG representatives assigned to OPTEVFOR is 
less than desired. 

The shortage of analysts precludes utilizing an 
analyst's expertise to design each of the test plans; 
although the analyst may try to review each test plan prior 
to its execution. In practice the majority of OPTEVFOR 
test plans are constructed by going through the files and 
modifying/copying a previously executed test plan. As a 
result, many tests are inadequately planned. 

Many times the analyst is not consulted until after 
the test has been completed, and he is then asked to 
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determine what the data show. This is inadequate, since 
the test has to be designed to obtain data in a fashion 
that will permit data analysis from which valid conclusions 
can be drawn. However, even OPTEVFOR's Project Instruction 
Volume II [Ref. 24], which is a comprehensive handbook for 
conducting OT&E and analyzing the data, essentially overlooks 
this essential part of T&E, the design of the experiment . 

As a result of the lack of an adequate number of 
capable analysts to design the tests and then analyze the 
data, needless expenses are often incurred, and incorrect 
conclusions can easily result. An example of needless 
expense is the recent OPTEVFOR testing of a missile which 
was supposed to demonstrate a kill probability of 0.4. 

Fifteen missiles were obtained to conduct the tests and all 
fifteen were fired. The test personnel reasoned that if six 
or more "hits" were obtained, a kill probability of 0.4 or 
greater would have been demonstrated. (6 hits/15 Missile 
Firings = Hit or Kill Probability Pk = 0.4.) A simple 
sequential test could have been designed where the firing 
of each missile was dependent upon the results of the 
previous firings. After each firing one of the following 
decisions could be made: accept the hypothesis that Pk is 
greater than or equal to 0.4, reject the hypothesis, or fire 
another missile. As an illustration of this test plan assume 
the first six missiles resulted in misses. This would 
demonstrate at a 95 percent confidence level that the kill 
probability is less than the required 0.4. This result is 
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obtained by using the binomial distribution with six failures 
out of six attempts and a test hypothesis that Pk is greater 
than or equal to 0.4. In this case, it would be a needless 
expense to fire any more than six missiles. If all fifteen 
missiles were fired, the last nine would essentially be 
wasted, since no greater statistical significance would need 
to be achieved. The actual missile test resulted in some 
"hits," but more missiles were fired than was necessary to 
reject the test hypothesis that the Pk was greater than or 
equal to 0.4. 

9 . T&E, Not Always Treated as a Continuing Process 
In order for T&E results to be used to accurately 
predict or assess a weapon system's operational effective- 
ness, continuous monitoring of the system's development 
progress must be maintained. Snapshot looks at the weapon 
system at infrequent intervals will not give a valid assess- 
ment of a system's potential. 

Making accurate assessments of a system's military 
worth and operational effectiveness requires considerable 
T&E experience on the part of the evaluator. As an illustra- 
tion of the need to monitor continually the development 
progress of a weapon system, this section will briefly 
describe the availability growth that should be tracked in 
every development program. 

Operational Availability (Ao) is defined as the 
ratio of the total time the system is capable of performing 
its function (uptime) to the total time when there is a 
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demand for the system (uptime plus total down time which 
includes maintenance/logistic delay times). Availability 
can also be expressed as "the probability that the system 
will operate satisfactorily at any point in time when used 
under stated conditions" [Ref. 25]. 

Mathematically operational availability is defined 
as follows: 



Ao 



MTBM 

MTBF + MDT 



where: MTBM = mean-time-between-maintenance 

(includes corrective and preventative 
maintenance ) 

MDT = mean-down-time including maintenance 
and logistic delay times 



Figure 6 is an illustrative time history of a system's 
operational availability growth during its development and 
initial deployment. Typically, the operational availability 
will not be a linear function. 

The evaluator needs considerable experience to 
accurately interpret such growth time histories. In the 
included example the dip in the curve after commencement of 
military T&E is typical and is usually caused by the 
problems attendant with the new operator developing his 
learning curve in operating the system. The learning curve 
phenomenon occurs again after the system is introduced to 
the fleet and a similar dip in operational availability is 
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is usually encountered. In this case increased system 
familiarity by the fleet results in more critical demands 
on system performance and consequently more deficiencies/ 
failures in the system will be detected. The rapid usage 
of the spares reduces the level of spares to the point that 
it adversely affects operational availability. This example 
illustrates why frequent tracking of operational availability 
rather than occasional snapshot looks is required to accurately 
evaluate a system's operational availability growth. This 
also applies to the tracking of a system's other operational 
suitability and performance parameters. 

C. GOOD FEATURES OF THE NAVAL AIRCRAFT T&E PROCESS 

Although numerous deficiencies exist in the Navy's T&E 
process for aircraft weapon systems, many good features 
exist : 

(1) Navy test pilots and NFO's (Naval Flight Officers) 
typically have and maintain excellent operational 
backgrounds 

(2) Testing is conducted by relatively small organizations 

(3) Competition among test agencies stimulates good 
performance 

(4) Decentralized test management Is practiced 

(5) T&E attracts top caliber military personnel. 

The Navy's developmental and operational test pilots and 
NFO's are required to have recent operational experience 
before receiving orders to a T&E project officer's billet. 
Pilots and NFO's are selected for admission to the Navy Test 
Pilot School on the basis of their operational, academic, 
and flight backgrounds. After graduation the engineering 
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test pilot or test flight officer is assigned to a project 
officer’s billet in one of the T&E Divisions at NATO, or 
occasionally NMC. Within three years the pilot or NFO is 
ordered back to an operational billet. In this way, the 
Navy keeps bringing current operational experience into its 
DT&E community, and test pilots do not remain in test 
billets for extended periods of time, so that they do not 
lose their operational perspective. The Air Force operates 
considerably different in that many of their test pilots 
spend the majority of their flying careers assigned to T&E 
billets, and they do not get re-exposure to operational 
environments. (The Viet Nam crisis somewhat changed this 
in that most Air Force test pilots were given one year tours 

of duty in Southeast Asia) . 

\ 

Navy aircraft T&E is conducted by relatively small 
organizations where the project officer has ready access to 
the Commander of the test activity. The so called "can do" 
attitude is evident in each of the T&E activities. This 
attitude is generally easier to develop in a small organiza- 
tion than in a larger bureaucratic organization. 

Since many T&E activities have similar capabilities, 
competition is generated between these activities in trying 
to solicit more project work, obtain greater test capabili- 
ties, -etc. This competition assists in causing each activity 
to become more cohesive, more work and task oriented, and 
better organized. Loyalty and pride in one's test activity 



78 



is prevalent and further facilitates good performance from 
a test activity. 

The project officer and- his team plan and conduct 
individual tests. High level T&E management develops and 
approves the TEMP and other broad T&E guidelines, but within 
these bounds the project officer is given essentially a free 
hand in the development of his test plan. This decentralized 
management in planning and conducting the tests causes the 
personnel conducting the test to be more responsible for the 
timely and effective accomplishment of the test objective. 

The Navy has made aircraft T&E a challenging, interesting, 
and rewarding job assignment. Consequently, the Navy has 
been able to constantly attract the highest caliber personnel 
for jobs in the DT&E agencies and OPTEVFOR. A tour of duty 
in a T&E Agency is regarded throughout the Naval Aviation 
community as career enhancing, and this further motivates 
good people to seek tours of duty in T&E. 

D. A PREDICTION OP THE EVOLUTION OP T&E BASED ON NO 

MAJOR CHANGES TO THE CURRENT T&E PROCESS 

Unless major changes are made to the Navy's current T&E 
process, a gradual and costly evolution will occur, and the 
roles of the T&E agencies will be altered significantly. A 
basic premise of this thesis is that if the Navy can define 
an optimal T&E process, it should make whatever changes are 
necessary to achieve this process, rather than relying on a 
slow evolutionary process which may or may not yield a 
desirable end result. 
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Previous sections of this thesis describe how the role 
of OPTEVFOR has become more encompassing and how this is 
causing undesirable duplication among test agencies. This 
appears to be the start of the evolutionary process that 
is predicted herein. 



Since OSD and DD(T&E) place primary credibility on T&E 
results obtained from OPTEVFOR, Navy Program Managers may 
soon feel compelled to task OPTEVFOR to participate in all 
Naval aircraft T&E. If this occurs, when a Program Manager 
has a system change that needs military T&E, he will 
probably request that only OPTEVFOR conduct the T&E. The 
Program Manager could have both a DT&E agency and OPTEVFOR 
conduct the T&E, or he might even try to arrange to have 
a joint OPTEVFOR/DT&E agency team conduct the evaluation. 
However, both these alternatives would cost more and take 
more time, since two T&E agencies vice one will have to 
analyze the data and report their results. So the Program 
Manager will make the cost-effective decision and have only 
one agency, OPTEVFOR, conduct the evaluation. Since the 
DT&E agency works for the developing agency, the Program 
Manager should have little trouble in squelching the dissen- 
sion that comes from the neglected DT&E agency. 

As this process continues the DT&E agencies might be 
forced to cut back personnel due to decreased work loads. 
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while OPTEVFOR increases its size. OPTEVFOR might be forced 
to establish its own test pilot school to teach the discipline 



of T&E. Soon OPTEVFOR will have technical capabilities 
that make it the only one capable of conducting many types 
of T&E. 

This evolution would continue until concern arises over 
the highly technical T&E that is being conducted by OPTEVFOR. 
As a result a suggestion might appear to once again create a 
completely separate organization composed of military person- 
nel with good operational backgrounds. These personnel would 
conduct T&E to determine if a weapon system is suitable for 
service use and to develop the optimum employment tactics. 
This new type of testing might be called Tactical Test and 
Evaluation (TT&E). 

An examination of the current T&E environment shows that 
such an evolution is well underway: 

OPTEVFOR is getting more and more projects. 

OPTEVFOR' s Personnel complement is increasing. 

More technical expertise is being obtained by OPTEVFOR. 

OPTEVFOR is being given more responsibilities while the 

DT&E agencies remain fairly stagnant. 

The need for an OT&E School is becoming more apparent. 

The need for a larger OPTEVFOR civilian cadre is becoming 

more apparent . 
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V. PROPOSAL FOR A TEST AND EVALUATION 
ORGANIZATIONAL CHANGE 



Changes are constantly being made in order to improve 
Naval aircraft T&E, but the thought of a major reorganization 
of the T&E activities has always been met with a high degree 
of apprehension. However, many problems facing Naval air- 
craft T&E will not be eliminated unless a major reorganiza- 
tion takes place in the T&E community. The less than 
optimal utilization of the Navy's T&E resources has been 
depicted, and many of the causes have been cited. Since 
T&E agencies having similar responsibilities report to 
completely separate commands, it is reasonable to assume 
that inefficiencies in terms of duplication will be diffi- 
cult to reduce. The belief that "more testing is better, 
so let's continue to do more and not worry so much about 
the duplication" is a simplistic and costly solution to the 
Navy's T&E problems, but one that has gained considerable 
support . 

A proposed approach to improving the Navy's T&E of air- 
craft weapon systems, while achieving an optimal allocation 
of the Navy's T&E resources, is the creation of a Naval 
Test and Evaluation Command that would have responsibility 
for all DT&E and OT&E conducted by the Navy with the excep- 
tion of the necessary T&E conducted by Navy Laboratories in 
fulfilling their research and development mission. The 
Commander of the Naval T&E Command (NTEC) should be a Vice 
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Admiral who also serves as a Director, Major Staff Office 
(DMSO) within OPNAV with the title of Director, Naval Test 
and Evaluation. Figure 7 shows this proposed OPNAV struc- 
ture. It should be noted that the dotted line relationship 
between the Chief of Naval Material and the Director, Naval 
T&E enables the Naval Material Command to assign DT&E and 
OT&E projects to the Naval T&E Command. Projects requiring 
Navy T&E from NAVAIR will be managed by the Chief of Naval 
Air T&E, and T&E projects from NAVSHIPS (Naval Ships Systems 
Command) will be managed by the Chief of Naval Ship T&E. 
Those T&E projects coming from the other systems commands 
of NAVMAT (Naval Electronic System Command, Naval Ordnance 
Systems Command, or Naval Facilities Engineering Command) 
or from NAVMAT designated Program Managers will be managed 
by either the Chief of Naval Air T&E or the Chief of Naval 
Ship T&E depending upon the nature of the project. 

The proposed organizational change would put the Direc- 
tor, Naval Test and Evaluation (DNT&E) in a newly created 
Director, Major Staff Office billet (OP-09A) reporting 
directly to OP-09, the Vice Chief of Naval Operations 
(VCNO). For compliance with DOD Directive 5000.3, the 
Director, Naval T&E, would report the results of his Naval 
T&E Command's evaluations directly to CNO, the immediate 
superior of the VCNO. As a DMSO directly under OP-09, the 
Director, Naval T&E would be able to maintain the necessary 
degree of independence from the developing agency that 
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PROPOSED OPNAV STRUCTURE, INCLUDING A DIRECTOR, NAVAL TEST 

AND EVALUATION 

FIGURE 7 
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should be required for the Commander of a T&E organization. 
He would never have to function both as the weapon system 
sponsor/proponent and as the T&E spokesman to the VCNO, as 
is now the case for OP-098. The data and judgments from 
each DMSO reporting to the VCNO would be evaluated as they 
are today. However added to the inputs from the other 
Major Staff Offices would be the Navy’s unfiltered input 
from the independent T&E command. This added voice to the 
VCNO should facilitate the making of better system 
acquisition decisions at the VCNO/CNO level. 

The creation of a DMSO for Naval T&E would not separate 
T&E from RDT&E. Development test and evaluation must be 
an integral part of the iterative research and development 
(R&D) process of design-test-evaluate-redesign that con- 
tinues until technical uncertainties and development prob- 
lems are resolved. This must, therefore, be the responsi- 
bility of the development contractor and the developing 
agency. The Director, RDT&E (OP-O98) would remain the CNO 
focal point for this activity and for all R&D that is occur- 
ring within the Navy or under Navy's funding. 

However, Navy conducted T&E would be organizationally 
separated from this process and would no longer be conducted 
by agencies working either for the program sponsor or the 
developing agency. This would place Navy conducted T&E 
in an ideal location, organizationally, independent from 
the developer and fully capable of reporting unbiased test 
results to the highest decision making level within the Navy. 
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A. DESCRIPTION OP PROPOSED T&E COMMAND 

The proposed Naval T&E Command (Fig. 8) would have two 
deputies reporting directly to the Director, Naval Test 
and Evaluation. One deputy, the Chief of Naval Aircraft 
Test and Evaluation would be responsible for all Navy T&E 
of aircraft and the associated weapons and ancillary equip- 
ment. The other deputy, the Chief of Naval Ship Test and 
Evaluation, would be responsible for all surface and sub- 
surface ship T&E and the associated weapons and ancillary 
equipment. No discussion is contained in this thesis on 
the organization or responsibilities of the proposed Chief 
of Naval Ship Test and Evaluation. It is recommended that 
a separate study be conducted in this area to evaluate the 
feasibility of such an organization and to determine a 
suitable composition for such an organization. 

The staff of the Chief of Naval Aircraft Test and 
Evaluation (CNAT&E) would be composed of personnel who would 
exclusively monitor and provide guidance for Range coor- 
dination, operational T&E by the VX squadrons, education 
of project personnel by either the Test Pilot School (TPS) 
or the System T&E Training Schools (STETS) to be created 
at each Test Center, and the typical staff functions of 
every major staff. The Commanders of the test centers 
and one test facility would report to the CNAT&E. 

1 . Naval Air Test Center, Patuxent River, Maryland 
The Naval Air Test Center at Patuxent River would 
be responsible for the complete T&E of the aircraft vehicle 
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PROPOSED TEST AND EVALUATION COMMAND 





(i.e. less the weapon system) and the associated auxiliary 
equipments. NATO would also become the air ASW test center, 
and would in addition perform T&E on most support aircraft. 
NATC would be composed of four test divisions and two train- 
ing divisions. No consideration is given in this thesis 
to an instrumentation or technical services division which 
would probably be required at each major test center. The 
Aircraft Test Division would absorb the responsibilities 
of the current Flight Test and Service Test Divisions. 
Aircraft Test would be responsible for the flying qualities, 
performance, and carrier suitability testing of fixed and 
rotary wing aircraft (Flight Test's current responsibilities) 
and for the developmental T&E of propulsion and aircraft 
systems, excluding electrical, armament, and avionics 
(Service Test's current responsibilities). 

The Naval Air Test Facility at Lakehurst, New Jersey 
would retain its responsibility for the T&E of catapults, 
arresting gear, and other aircraft launch and recovery 

equipments. Having NATF work directly for NATC would more 

\ 

easily facilitate better utilization of the catapult and 
arresting gear facilities that are located at both Patuxent 
River and Lakehurst. 

The Anti-Submarine Warfare Systems Test (ASWST) Division 
would replace the current Weapons System Test Division at 
NATC. ASWST would be responsible for the developmental 
T&E of the electrical armament, and avionics systems in all 
rotary wing and fixed wing ASW aircraft and in support 
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aircraft, such as AEW (airborne early warning) aircraft and 
cargo aircraft. VX-1 would retain its present role of OT&E 
of ASW aircraft systems but would also become responsible 
for the OT&E of certain support aircraft. VX-1 would main- 
tain its operational type organization and its close rela- 
tionship with the operational forces. 

The U.S. Naval Test Pilot School (TPS) would con- 
tinue to report to the Commander, NATC, and its functions 
would remain unchanged. Approximately half of its Naval 
officer graduates would become project officers in the 
expanded Aircraft Test Division, and the remainder would 
be assigned to project officer billets primarily at the 
systems test divisions at the three test centers. While 
TPS would remain a relatively long duration school (one year) 
and would provide T&E training to all of the Aircraft Test 
Division's project officers and civilian engineers it would 
only provide training to a relatively small number of the 
project officers and civilian engineers throughout the rest 
of the Naval Aircraft T&E Command. Therefore, schools of 
relatively short duration, called System Test and Evaluation 
Training Schools (STETS), would be established at each test 
center to teach the fundamentals of T&E and provide a basic 
foundation in the technologies utilized in the necessary 
warfare specialities; i.e., ASW, air-to-air, attack, ECM, 
etc. (Appendix F presents the basic objectives of these 
schools). A STETS would be established at the Naval Air 
Test Center to teach the fundamentals of ASW and of any 
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required support warfare to the project officers and civilian 
engineers/analysts reporting to VX-1 and to the ASW Systems 
Test Division. Officers that just graduated from TPS might 
require a brief supplementary course from the STETS prior to 
commencing work in the systems test division, and those 
officers coming back for second tours of duty in T&E might 
require a special refresher STETS course. 

2 . Naval Fighter Test Center, Pt . Mugu, California 

The Naval Missile Center Complex at Point Mugu would 
become the Naval Fighter Test Center and be responsible for 
the T&E of all air to air weapons systems and ancillary and 
support equipment. The Naval Fighter Test Center would be 
composed of basically four divisions: a Fighter Systems 

Test (FST) Division, VX-*} , Fighter System T&E Training 
School (FSTETS), and PMR (Pacific Missile Range) Division. 

FST would be responsible for the developmental T&E 
of fighter aircraft weapons, avionics, and electrical 
systems, and VX-4 would be responsible for the OT&E of the 
complete fighter aircraft system and/or air to air weapon 
system. The Fighter STETS would provide a relatively short 
course in T&E of fighter weapon systems to project officers 
and engineers/analysts reporting to FST and VX-*1. It would 
function similar to the STETS at the Naval Air Test Center. 

The PMR division would be responsible for the opera- 
tion and management of the Pacific Missile Range. 
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3 . Naval Attack System Test Center, China Lake, 

California 

The China Lake complex (excluding the Naval Weapons 
Center'*') would become known as the Naval Attack Systems 
Test Center (NASTC). It would be responsible for the T&E 
of all attack aircraft weapon systems, and all ECM systems 
and reconnaissance systems (excluding ASW) . The center's 
responsibilities would include DT&E of the aircraft armament, 
avionics and electrical systems of all attack, ECM, and 
reconnaissance aircraft, the OT&E of these aircraft, and the 
complete Navy T&E of all air to surface weapons. The NASTC 
would be organized similar to the NFSTC at Pt . Mugu. 

The four basic divisions of the NASTC would be 
Attack Systems Test (AST), VX- 5 , Attack STETS, and Range 
Division. The AST would be responsible for the developmental 
T&E of all attack aircraft weapon systems and weapons, and 
VX-5 would be responsible for the OT&E of the complete attack 
aircraft and/or air to surface weapon. The Attack STETS 
would provide the necessary attack, ECM, and reconnaissance 
systems T&E training for VX-5 and AST project officers and 
engineers/analysts. The Range division would be responsible 
for the operation and management of the various China Lake 
ranges . 

^The Naval Weapons Center (NWC) which is a Navy research 
and development laboratory located at China Lake would • 
continue to report to the Chief of Naval Research and be the 
principal RDT&E Center for air warfare and missile systems. 
NWC would be able to task the NASTC to conduct DT&E and 
OT&E. 



4 . Naval Weapons Evaluation Facility (NWEF), 
Albuquerque, New Mexico 

NWEF would retain Its current function of T&E of 
nuclear weapons and the associated systems within the 
compatible delivery aircraft. NWEF would be subdivided 
into a technical division. Nuclear Weapons Systems Test, 
and an OT&E division, and a VX squadron. As a result of the 
relatively small size of NWEF a formal STETS would not be 
established. Instead its project officers and engineers/ 
analysts would receive T&E training at the Attack STETS at 
China Lake and the necessary nuclear technical training 
through an informal NWEF training course. 

B. DT&E PERSPECTIVE 

Discussion of the proposed reorganization with personnel 
in the Naval Air Systems Command was usually met with dis- 
sension. Their belief that the developing agency requires 
absolute control over all DT&E is unfounded. Most aircraft 
DT&E is conducted by the contractor, and it is only that 
portion of DT&E conducted by the Navy that is in dispute. 
Under the concept proposed in this thesis, the developing 
agency could task the Naval T&E Command to perform whatever 
type T&E is required. The funding for the T&E would be 
provided by the developing agency, and the Naval Air T&E 
Command would report its T&E results to the developer, 
exactly as is done today. 

From the developer's perspective little change will 
have actually occurred other than the DT&E agencies would 
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no longer work directly for the developer. The Program 
Manager, the key man in managing the development and produc- 
tion of a weapon system, would operate in the NAVMAT Matrix 
organization in the same way as he does today. The PM 
would still task the various functional NAVAIR Divisions, 
other System Command Divisions, and civilian contractors 
to perform essential functions for his program. However, 
instead of tasking a NAVAIR agency to perform DT&E and 
OPTEVFOR to perform OT&E, he would task the Naval T&E 
Command to perform all of his Navy T&E. 

The developing agency could task the Naval Air T&E 
Command to conduct only test and analysis rather than test 
and evaluation. This would permit the developing agency 
to merely task the T&E Command to obtain specific test data, 
but not to evaluate the significance of the data. This 
type of request from the developing agency might occur if 
the fleet were to lose one or more aircraft for an unexplained 
reason (e.g. , the wing falling off in flight, but all inci- 
dents occurring over water and, therefore, no evidence exists 
which would point to the cause). In such a case NAVAIR 
would probably task the Naval Air T&E Command to perform 
numerous structural tests and analysis and merely send the 
various load data to NAVAIR for its evaluation, which would 
probably be done jointly with the particular airframe 
manufacturer . 

At present, much of the development testing is planned 
by the Naval Air System Command, and very detailed 
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descriptions of the tests and test conditions are given to 
the DT&E agencies in the AIRTASKS, the official NAVAIR 
document which assigns work and provides funding to the 
DT&E agencies under NAVAIR* s control. This same type 
procedure should work when NAVAIR tasks the T&E Command to 
perform development tests. 

With Congressional pressure to decrease the military's 
Washington D.C. personnel, NAVAIR is facing severe cutbacks 
which could have an adverse effect on maintaining T&E 
expertise. The creation of a T&E Command would provide 
a simple means which would permit NAVAIR to decentralize 
this expertise. Many NAVAIR engineers could be transferred 
to the T&E Command which would accomplish two objectives: 

(1) It would decrease the number of Navy personnel 
in Washington D.C. 

(2) It would place the majority of the Navy's aircraft 
T&E expertise in the test agency. 

In the current system much of the Navy's T&E expertise 

exists in NAVAIR, and consequently NAVAIR does much of 

the test evaluation, and the test centers do too little 

actual test evaluation. 

Since the development agency is responsible for develop- 
ing a system that works, meets specifications, and is opera- 
tionally suitable, the timely and effective conduct of OT&E 
is as much a responsibility of the Program Manager as is 
the timely and effective conduct of DT&E. Under the proposed 
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T&E organizational concept, the development agency would 
be able to task the T&E command to perform OT&E in the same 
way as it would task the T&E Command to perform DT&E. 



At present the Program Manager is essentially the NAVAIR 



decision point on where to conduct various tests for pro- 
grams that are not tracked by DSARC . It is unlikely that 
each Program Office has sufficient T&E expertise to always 
make test location decisions which contribute to optimizing 
the utilization of the Navy's T&E resources. Program 
Manager's T&E decisions on where to conduct various tests 
are influenced unduly by the program's dollar and time con- 
straints ltfith too little regard for the expertise of the 
test agency. 

In the current T&E structure, the developing agency's 
Program Manager has three alternatives when he requires 




that OT&E be performed: 

(1) Task a DT&E agency to perform the tests and call 
the tests "testing for service suitability" vice 
OT&E 

(2) Generate a request to CNO to have a Development 
Assist-*- project assigned to OPTEVFOR to conduct the 
desired OT&E 

(3) Request that an operational unit (i.e. an aircraft 
squadron, ship, etc.) evaluate the item under 
operational conditions. 



■'"Development Assists are projects which are assigned 
by CNO at the request of the Developing Agency to OPTEVFOR 
[Ref. 26]. 
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In the proposed T&E organization the Program Manager 
would no longer have the flexibility in deciding which 
organization should do Navy T&E. The Naval T&E Command would 
make that decision and be expected to optimize the utiliza- 
tion of the Navy’s T&E resources while obtaining credible 
T&E results. 

One criticism of the proposed change has been the size 
of the testing command that would be created and the many 
problems that are typical of large organizations. Care must 
be exercised to minimize creation of "paper work" stumbling 
blocks which are so easy to generate in large organizations. 
Prior to the establishment of a T&E Command, a study should 
be conducted to determine the best way to organize such a 
command while maintaining testing efficiency, especially in 
regard to timely completion of assigned T&E including 
publishing the reports. 

One of the major problems facing Program Managers is 
obtaining T&E results within the time constraints specified 
in the tasking document (e.g., AIRTASK). As discussed in 
Section IV. B. 6, the current T&E organization is frequently 
guilty of late completion of T&E reports. A fear voiced by 
Program Managers regarding the creation of a T&E command is 
the potential problem of its unwillingness to perform T&E 
in accordance with the Program Manager's time requirements 
with the Program Manager having no capability to order the 
T&E agency to perform as directed. Theoretically, this so 
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called "ordering" can be done now, but in practice this 
requires a dialogue between the Commander, Naval Air 
Systems Command and the Commander of the test agency. The 
decision as to whether to comply with the Program Manager's 
urgent T&E requirement will then consider the overall agency 
T&E workload and NAVAIR's overall requirements. In the 
proposed structure the Program Manager would hve to resort 
to different means to induce this same type of high level 
dialogue to determine the relative priority of his project. 
However, the central control of T&E should reduce the 
problem of determining relative project priorities, since 
random inputs from CNO to accomplish "his" special project 
would be much less likely, and these projects often cause 
arbitrary priorities. 

An examination today of the responsiveness of OPTEVFOR 
and the timeliness of its completing T&E and publishing 
reports should dispel any doubts that an "independent" T&E 
agency would respond as rapidly for the developer as a T&E 
agency that works for the developer. OPTEVEOR does respond 
as rapidly and publishes its reports probably more promptly 
than the NAVAIR controlled DT&E agencies. 

C. OT&E PERSPECTIVE 

One of the big concerns of OPTEVFOR is the need to 
maintain a close identity with the operational community. 
There is a concern that if OT&E becomes too technical, it 
can no longer be truly operational. By adopting the 



proposed T&E organizational change, the VX squadrons would 
be better able to maintain their valuable operational 
perspective, since they would not be compelled to develop 
an imbalance of technical expertise over operational 
expertise as appears to be happening today. Since the VX 
squadrons are being tasked to conduct some T&E solely on 
the basis of their "independence , ” rather than their 
expertise, OPTEVFOR is compelled to become more technically 
oriented in order to perform capably the increased technical 
T&E requirements. The danger of having this continue to the 
point that OPTEVFOR develops an overbalance on the technical 
aspects of T&E should be obvious. 

The concept of having the VX squadrons operate very much 
like an operational squadron in order to maintain a better 
operational perspective would be retained. The VX squadrons 
would work closely with the co-located systems test division 
(ASW, fighter, or attack) throughout an aircraft’s develop- 
ment. T&E reports from each of the test centers could 
contain solely the results of DT&E or OT&E or the reports 
could contain the results of combined testing (DT&E and 
OT&E). The combined T&E reports would have a technical 
section (for DT&E) and an operational section (for OT&E), 
and differing opinions in each section could occasionally 
be expected. However, the CNAT&E staff would comment on 
reports having significantly different findings in the 
technical and operational sections in order to aid DD(T&E), 
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DSARC, Program Manager, or other appropriate decision 
makers in reconciling the different opinions. 

In the past the biggest contribution made by the VX 
squadrons was in the area of tactics development. This 
very important area cannot be overlooked, and the proposed 
organization would ensure that tactics development remains 
a major function of each VX squadron. By having the VX 
squadrons involved earlier in weapon system development they 
would be better prepared- to develop sound tactics for each 
weapon system much sooner than in the past. (Previously 
it often took more than a year of fleet operation with a 
weapon system before a VX squadron released its report on 
optimum use procedures and employment tactics). 

When personnel are involved in finding deficiencies and 
recommending changes to a weapon system throughout its 
development cycle, they very often develop strong feelings 
against certain features which they reported as being defi- 
cient, and which are often not corrected. It is important 
to have the people who are developing tactics for a weapon 
system fully believe in and be strong proponents of the 
potential of that weapon system. This is a potential prob- 
lem area that exists in the current T&E process where the 
OT&E project officers are being put more into the role of 
developmental testing, in that they perform tests during 
the various iterations of design, and the specific defi- 
ciencies they cite assist in the evolution of the design. 
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This type of involvement has often caused a DT&E project 
officer to develop strong dissatisfactions toward the weapon 
system he tested. 

The proposed T&E organization would not put the OT&E 
project officer into this design iteration testing process, 
but would allow him to view more readily the development 
of the entire system and not each subsystem design iteration. 
This will enable the OT&E project officer to maintain a 
better operational perspective of the entire system while 
becoming involved with the system early enough to develop 
tactics on a timely basis. 

Perhaps the most overlooked, but often most valuable, 
portion of OT&E is the de facto OT&E that is conducted 
during the early deployments of a weapon system. As hard 
as the Navy tries to move OT&E earlier into the development 
cycle, it can never eliminate the value of the actual de- 
ployment of the weapon system in the operational environ- 
ment. At present the operational command using the system 
is often confused as to whom he should report system prob- 
lems. NAVAIR is naturally cognizant for aircraft systems 
but various T&E agencies often would be able to make a rapid 
and valuable contribution if they were informed of the 
problem. The user consequently informs NAVAIR but is usually 
uncertain as to whether to also contact OPTEVFOR, the appro- 
priate VX squadron, NATC , NWC , NMC, or NWEF. The proposed 
organization would facilitate better retrieval of operational 
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data from the fleet by eliminating the operational command's 
dilemma in trying to determine which T&E agency is cognizant 
of each problem that arises. 

J 

D. T&E ORGANIZATIONAL CHANGE WILL CORRECT CURRENT T&E 
DEFICIENCIES 

Many of the T&E deficiencies cited in Section IV of 
this thesis would be corrected by the adoption of the pro- 
posed T&E organizational change as follows: 

1 . Organizational Deficiencies 

The creation of a DMSO for Naval T&E would ensure 
that proper high level attention is given to independent 
T&E results. Since the Director, Naval Test and Evaluation 
would not be assigned as a program sponsor, he would not 
be forced to change hats between the user and the producer 
(advocacy) roles as now occurs in OPNAV with OP-098. 

Having the Director, Naval T&E report to 0P-09 
would preserve the test organization independence that is 
required by directive. This independence also appears 
logical and makes intuitive sense for a test organization. 

In many aircraft companies the Chief Test Pilot reports 
to the president as often does the Product Assurance Divi- 
sion.'*' The entire T&E command can be thought of as a 



^"Product Assurance is the term now given to many com- 
panies' previous Quality Assurance Divisions, since Product 
Assurance is meant to be more encompassing. Product assur- 
ance includes the disciplines of design assurance, maintain- 
ability, reliability, test and evaluation, and safety. 
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Product Assurance group that ensures weapon systems meet 
specifications and are operationally suitable. Reporting 
of such an organization to a high level ensures unfiltered 
test results. 

Duplication of T&E responsibilities would be more 
easily reduced if all Navy T&E is controlled within one 
Command. The T&E Command would install among the various 
DT&E agencies central line management control, which 
doesn't exist today in the NAVAIR T&E coordinator. 

Since only one T&E Command would exist in the Navy, 
the flexibility of where to assign T&E projects would no 
longer rest with the Program Manager, but with the T&E 
Command. Th is w ould help ensure that adequate T&E is 

/O 

performed on non-major procurements. 

The creation of a T&E Command which is entirely 
independent from the developing agencies would eliminate 
the dilemma facing DD(T&E) today. He would be able to place 
equal credibility on all test results from the T&E Command, 
and would not have to request an operational test agency 
to repeat a technical test merely to ensure that the re- 
quired degree of independence is maintained in the test 
results . 

Placing all T&E agencies under one command would 

/ 

facilitate the creation of a uniform system of funding 
throughout the testing community. It would also simplify 
the task of identifying all the funding required for T&E. 
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With all the test agencies under one command, all 
ranges would also be controlled by the same command. This 
central control of ranges would facilitate easier identifi- 
cation of effective and efficient range use and funding 
requirements . 

2 . Testing Deficiencies 

Much of the overlap between OT&E and DT&E within 
the Navy has been caused by the need for independence being 
identified with only OT&E. By having all T&E independent 
of the developing agency the different perspectives of the 
OT&E and DT&E roles can be more easily retained. 

Having T&E Command visibility at the VCNO level 
would ensure that test planning is given more attention 
and T&E experts become involved in this planning. This 
high level visibility should ensure the creation of better 
operational test objectives. 

The problem of inadequate technical expertise in 
the OT&E test agency, OPTEVFOR, would be somewhat reduced 
as a result of the elimination of the need for the OT&E 
agency to expand its testing expertise into more developmen- 
tal type testing. The creating of STETS (System T&E Train- 
ing Schools) at each test center would provide a place to 
give all project officers and civilian engineers/analysts 
a basic background in T&E, and should therefore improve 
the entire OT&E and DT&E process. 

Late test results is a problem that would not be 
corrected by the proposed organization alone. Since each 
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test center would be organized similarly, each test center 
would be motivated to develop the most efficient T&E 
report review process. However, this area should be given 
considerable attention when creating the Naval T&E Command. 

With a T&E Command that is independent from the 
developer and that is capable of ensuring specification 
compliance, there would no longer be a need for BIS. It 
is, therefore, recommended that BIS be disestablished if 
this thesis proposal is adopted. 

By having the VX squadrons within the T&E Command, 
the need for analysts throughout the T&E Command would become 
obvious. As the need for more analysts within the T&E 
community is recognized a better career pattern for analysts 
within Navy T&E should evolve. This would increase the 
ability of Navy T&E to attract and retain high caliber 
analysts . 

By having one T&E Command it will be easier to 
ensure continual monitoring of development programs. The 
infrequent snap shot looks at a program by each of the Navy 
agencies which often occurs today was shown to be an unsat- 
isfactory method for assessing system potential. The pro- 
cedure of assigning a resident T&E team composed of both 
DT&E and OT&E personnel during a systems' early development 
is being utilized by the Air Force on the B-l program 
[Ref. 27] and was utilized on the F-15 program. This 
procedure has considerable merit and will be relatively 
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easy to implement if all Navy T&E personnel were assigned 
to one Naval T&E Command. 

The creation of a Naval T&E Command will give 
increased stature to all T&E within the Navy. This should 
make T&E duty for Naval Officers more highly desirable for 
second and third tours of duty than it is today. Effective 
military T&E can only be planned and accomplished by per- 
sonnel who have operational and technical backgrounds and 
testing experience. Too little emphasis in the past has 
been placed upon encouraging good officers to become 
specialized in the T&E field and to develop that necessary 
testing experience. As a result, many middle and top mana- 
gers within the T&E agencies are not of the caliber necessary 
to manage the type of effective and efficient T&E that the 
Navy's limited resources demand. The creation of a Naval 
T&E Command will hopefully do much to eliminate this 
deficiency . 



VI. CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

1. The valuable internal "user-producer dialogue" is 
lost during the latter stages of a weapon system's develop- 
ment, since the Navy's independent T&E organization, OPTEVFOR, 
reports to OP-O 98 who is compelled to act as a system propo- 
nent and must often conceal weapon system development 
problems . 

2. Neither the development T&E agencies nor OPTEVFOR 
is independent from the influence of the developing agency. 

3. An excessive amount of duplication exists in the 
testing conducted by the major participants in the Navy 
aircraft weapon system T&E process. 

4. Inadequate central management exists among the NAVAIR 
T&E activities, consequently costly duplication of facilities 
and expertise exists among the NAVAIR T&E activities. 

5. OPTEVFOR is being tasked to perform T&E often on the 
basis of its independence rather than on its testing 
capability . 

6 . Adequate checks do not exist in the Navy's acquisi- 
tion process for non-major systems to ensure that adequate 
T&E is conducted and that the major deficiencies are 
corrected prior to procuring and deploying the system. 

7. The costs for conducting T&E cannot be totally 
identified, and numerous methods are utilized by the T&E 
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agencies to calculate the costs of their performing T&E. 

This lack of standard cost accounting makes T&E cost- 
effectiveness decisions nearly impossible to perform. 

8. Central aircraft T&E range management does not 
exist, consequently it is a difficult task to justify the 
various ranges and make rational decisions when forced to 
reduce range assets. 

9. A clear distinction between the responsibilities 
of DT&E and OT&E does not exist, consequently testing 
duplication is increasing. 

10. Test agencies do not participate adequately in 
early T&E planning. 

11'. . Adequate criteria against which OT&E can be 
conducted often does not exist. 

12. Inadequate technical expertise exists in OPTEVFOR 
to effectively perform all of its responsibilities. This 
problem has been magnified by the increased T&E responsibil- 
ity given to OPTEVFOR. 

13. A definite requirement exists for a technical course 
of instruction in the discipline of Weapon System T&E. 

1^. • Program Managers frequently do not receive test 
results from the various test agencies soon enough to assist 
in making vital program decisions. 

15. OPNAVINST 3960.8 shows BIS commencing along with 
OPEVAL prior to the DSARC III major production decision. 

This sequencing is illogical if BIS and OPEVAL are to retain 
their current roles. 
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16 . A shortage of statistical analysts exists throughout 
the T&E community (DT&E and OT&E). 

17. Test and evaluation is often treated as a process 
where snapshot looks at a system can give valid conclusions 
as to the system’s development progress. This is false, and 
T&E must be treated as a continuing process.. 

18. Many good characteristics continue to exist in the 
Navy’s T&E process. 

a) Navy test pilots and NFO's typically have and 
maintain excellent operational backgrounds 

b) Testing is conducted by relatively small 
organizations 

c) Competition among test agencies stimulates good 
performance 

d) Decentralized test management is practiced 

e) T&E attracts top caliber military personnel 

B . RECOMMENDATIONS 

1. The Navy should create a Naval T&E Command that 
would have responsibility for all Navy conducted T&E and 
OT&E with the exception of the necessary T&E conducted by 
Navy laboratories in fulfilling their R&D missions. 

2. The Commander of the Naval T&E Command should also 
serve as a Director, Major Staff Office reporting to OP-09, 
the VCNO . 

3. The Naval Air T&E Command should be organized as 
proposed in this thesis. 

ij . A study should be conducted to determine the 
appropriate organizational structure and functions of a 
Naval Ship T&E Command. 
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5. A System T&E Training School should be established 
at each of the major test centers. 



109 



APPENDIX A 



ORGANIZATIONS WITH WHICH INTERVIEWS WERE CONDUCTED 



/ 

1. Office of the Secretary of Defense, Washington, D.C. 

DDR&E (T&E ) 

2. Office of the Secretary of the Navy, Washington, D.C. 

ASN (R&D) 

3. Office of the Chief of Naval Operations, Washington, D.C. 

OP -983 

OP-506 

OP -508 



4. Naval Air Systems Command, Washington, D.C. 

AIR 05 
AIR 510 
PMA-241 
PMA-2M 

NAVAIR T&E Coordinator, Patuxent River, Md. 



5. Senate Armed Services Committee 
RDT&E Subcommittee 



6. Naval Air Test Center, Patuxent River, Md. 

7. Sub Board of Inspection and Survey, Patuxent River, Md. 



8. Operational Test and Evaluation Force 
Headquarters, Norfolk, Va. 

Air Development Squadrons One, Four and Five 
Deputy COMOPTEVFOR, North Island, Ca. 
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APPENDIX B 



DEFINITIONS OF DEVELOPMENT AND OPERATIONAL TEST AND EVALUATION 

Development Test and Evaluation (DT&E) 

DT&E is that T&E conducted under the sponsorship of the 

developing agency to facilitate in the evolution of the 

system. Its specific functions are to: 

"(1) demonstrate that the engineering design and development 
process is complete. 

(2) demonstrate that the design risks have been minimized. 

(3) demonstrate that the system will meet specifications. 

(*0 estimate the system's military utility when introduced." 
[Ref. 6 ] 

The types of tests included in DT&E are: engineering 

tests, contractor/laboratory demonstrations. Navy Technical 
Evaluations (NTE), Navy Preliminary Evaluations (NPE), and 
deficiency correction tests. Advanced development, engineer- 
ing and production prototype models are used for DT&E. 



Operational Test and Evaluation (OT&E) 



"OT&E is that test and evaluation conducted to estimate 
the prospective system's military utility, operational 
effectiveness, and operational suitability (including 
compatibility, interoperability, reliability, maintain- 
ability, and logistic and training requirements), and 
need for any modifications. In addition, OT&E provides 
information on organization, personnel requirements, 
doctrine, and tactics . . . OT&E will be accomplished 

by operational and support personnel of the type and 
qualifications of those expected to use and maintain 
the system when deployed, and will be conducted in as 
realistic an operational environment as possible." 

[Ref. 6] 



111 



OT&E is divided into two categories, initial operational 
test and evaluation (IOT&E) and follow-on operational test 
and evaluation (FOT&E). IOT&E is that OT&E conducted prior 
to the first major production decision (DSARC III). FOT&E 
is the continuing OT&E of a weapon system after the produc- 
tion decision has been made. 
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APPENDIX C 



INSTRUCTIONS/DIRECTIVES PERTAINING TO THE T&E OF NAVAL 

AIRCRAFT WEAPON SYSTEMS 



DOD Directive 5000.1, "Acquisition of Major Defense Systems", 
13 July 1971 

DOD Directive 5000. 3> "Test and Evaluation", 19 July 1973 

SECNAV Instruction 5000.1, "System Acquisition in the 
Department of the Navy", 13 March 1972 

OPNAV Instruction 3930. 8B "Assignment and Prosecution of 
Test and Evaluation Projects", 5 April 1973 

OPNAV Instruction 3960.8 "Test and Evaluation of Navy 
Systems and Equipments", 22 January 1973 

OPNAV Instruction 5^20.70, "Mission, Organization and 

Function of the Board of Inspection and Survey", 

2 April 1971 

OPNAV Instruction 5^0. ^7D, "Mission and Functions of 

Operational Test and Evaluation Force (OPTEVFOR)", 
6 March 1973 

NAVAIR Instruction 5^100.276, "Assignment of Coordination 
Responsibilities for the Naval Air Systems 
Command's Test and Evaluation Functions", 

20 December 1972 

INSURV Instruction 13100. 1A, "Aircraft Acceptance Trials; 
Procedures For", 1 October 1972 
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APPENDIX D 



NAVAL AIRCRAFT WEAPON SYSTEM T&E ORGANIZATIONS 



Naval Air Systems Command 

Naval Aerospace Recovery Facility, El Centro, Calif. 

Develop, test, and evaluate parachutes and related 
systems for human escape methods and systems, and 
retardation and recovery systems. 

Naval Air Engineering Center, Philadelphia, Pa. 

RDT&E in launching, recovery, and landing aids for 
aircraft and in ground support equipment for aircraft 
and airborne weapons systems. 

Naval Air Propulsion Test Center, Trenton, N.J. 

Test and evaluate aircraft propulsion systems — their 
components, accessories, fuels, and lubricants. 

Naval Air Test Center, Patuxent River, Md. 

Test and evaluation of aircraft weapons systems, 
their components and related equipment. 

Naval Missile Center, Pt . Mugu, Calif. 

Test, evaluation and development support of Naval 
weapons, weapons systems and related devices. 

Naval Weapons Evaluation Facility, Alburauerque , N.M. 

Test and evaluation and technical support for nuclear 
and designated non nuclear weapons and weapon systems. 

Operational Test and Evaluation Force 



VX-1 (Air Test and Evaluation Squadron One), Patuxent River, Md. 
OT&E of airborne anti-submarine weapon systems, support 
systems, components, and equipment. 

VX-4, Point Mugu, Calif. 

OT&E of all-weather fighter weapons systems and air- 
launched guided missiles weapon systems. 



VX-5, China Lake, Calif. 

OT&E of airborne attack weapon systems and support systems. 



Chief of Naval Research 



Naval Air Development Center, Warminster, Pa. 

Principal Navy Research, Development, T&E center for 
Naval Aircraft systems. 

Naval Weapons Center, China Lake, California 

Principal Navy RDT&E center for air warfare and 
missile systems. 



115 



APPENDIX E 





NAVY RANGES AND FACILITIES USED 




FOR AIRCRAFT WEAPON SYSTEM T&E 


Ranges : 





Atlantic Fleet Weapons Range, Roosevelt Roads, Puerto 
Rico ' 

Atlantic Undersea T&E Center (AUTEC), West Palm Beach, 
Fla. 

Pacific Missile Range (PMR), Pt . Mugu, Ca. 



Naval 


Weapons Center (NWC) Ranges, China Lake, Ca. 


Naval 


Weapons Laboratory (NWL) Ranges, Dahlgren, Va. 




S' 



Facilities with their associated test cites: 



Naval 


Aerospace Recovery Facility, El Centro, Ca. 


Naval 


Air Propulsion Test Center, Trenton, N.J. 


Naval 


Air Test Center (NATC) , Patuxent River, Md. 


Naval 


Air Test Facility (NATF), Lakehurst, N.J. 


Naval 


Explosive Ordnance Disposal Facility, Indian 
Head, Md. 


Naval 


Ordnance Missile Test Facility, White Sands, 

N.M. 


Naval 


Weapons Evaluation Facility (NWEF), Albuquerque, 
N.M. 
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APPENDIX F 



SYSTEM TEST AND EVALUATION TRAINING SCHOOLS (STETS) 

The main purpose of the STETS would be to teach the 
basic fundamentals of T&E to project officers and civilian 
engineers/analysts who are engaged in aircraft weapon system 
T&E. Specifically, each major test center would have a 
STETS which would provide courses of study for all project 
officers and civilian engineers/analysts who are being 
assigned to T&E jobs. Each test center's STETS would con- 
centrate on teaching the fundamentals of the particular 
warfare specialties of that test center (e.g. STETS China 
Lake would concentrate in the areas of aircraft attack 
systems, air to ground weapons, electronic countermeasures, 
and reconnaissance). 

In addition to teaching the technical and statistical 
aspects of T&E, STETS would also conduct classes dealing 
with the following: the test center's range/s utilization 

procedures and capabilities, the Navy's organization for 
weapon system acquisition and the role that the test center 
fulfills, the test center's funding system and the proce- 
dures that must be followed to enable it to function, 
technical services (instrumentation, photo coverage, 
computer time, etc.) that are available to the test team 
and how to obtain those services, and technical report 
writing. 
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The STETS full course of study would take approximately 
eight to ten weeks with half day attendance. The half day 
attendance would enable the pilots and NFO's to spend the 
other halves of these days In familiarizing themselves with 
the aircraft/new systems they would eventually be testing, 
in further developing their flying proficiency, and in 
commencing an orderly project takeover from the project 
officer they would be replacing. The civilian engineers/ 
analysts would spend the other halves of these days working 
in their respective divisions by providing assistance to 
the experienced engineers in data reduction or in some other 
tasks which they could fulfill. 

Besides the STETS full course of study, abbreviated 
courses should be offered. These abbreviated curriculums 
would serve as refresher courses for officers who are 
reassigned to T&E billets and have already graduated from 
the full STETS course. The abbreviated curriculum could 
also be used to give TPS graduates that are being assigned 
to non-aircraft division billets a better foundation in 
Weapon System testing. 

The following is a typical course content which could 
be utilized by the STETS. 

1. Mathematics/Mechanics refresher - introductory 
course including analytical geometry, calculus, statics, 
and dynamics. Its main purpose would be to ensure all 
students have a reasonable foundation in these essential 
engineering fundamentals. 
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2. Statistics - a course to give the student sufficient 
depth of understanding in probability and statistics that 

he would be able to appreciate the statistical implications 
of test design and test results. Course would present some 
simple statistical test designs and illustrate the increased 
test effectiveness and potential cost savings that are 
possible when tests are designed knowledgeably. 

3. Computer Technology - an introductory course where 
digital and analog systems are studied. Fundamentals would 
be studied and student would be taught how to utilize 
various computer systems available at the test center. The 
integration of computers into complex weapon systems would 
be studied as would methods of differentiating computer 
software problems from system hardware problems in complex 
weapon system testing. 

4. Fundamentals in Warfare Specialty (more than one of 

these might be offered at each STETS) - course would teach 
the basic fundamentals of the particular warfare specialty. 
General areas to be studied include: test techniques, sys- 

tem integration problems, state of the art limitations, 

new technological advances and their potential applications, 
etc . 

5. Data acquisiton - course would study different types 
of instrumentation and photo equipment systems. Capabili- 
ties, limitations, and relative expenses of each would be 
studied. Each student would learn how to read different 
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types of instrumentation and become familiar with the 
typical errors that may result. 

6. Weapon System Acquisition Process - this course 
would present a general overview of the entire acquisition 
process highlighting how the particular test center fits 
into the process. Requirements determination, DSARC pro- 
cess, DOD PPBS process, DOD agency roles and responsibili- 
ties, DOD budget cycle, program manager's role, etc. would 
be studied. 

7. Range Operation - operation and capabilities of the 
test center's ranges would be studied as would the capabil- 
ities available at other Navy and DOD ranges. The particu- 
lar operating procedures for the test center's ranges would 
be studied. 

8. Test Center Support - the myriad of support facili- 
ties at the test center would be studied as would the pro- 
cedures required to obtain these services. These include 
instrumentation, photo coverage, computer services, aircraft 
maintenance, etc. 

9. Project Funding - the operation of the test center's 
budget system would be studied as would the procedures 
required to obtain funding for a particular project. The 
test center's budget system reporting requirements would 

be taught . 

10. Report Writing - this course would teach technical 
report writing in accordance with the test center's 
procedures . 
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